[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:02 dist-test-slave-7zlw krb5kdc[25524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:02 dist-test-slave-7zlw krb5kdc[25524](info): set up 2 sockets
Sep 26 09:22:02 dist-test-slave-7zlw krb5kdc[25524](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 09:22:03.349807 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:03 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:03.517831 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46019
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:46019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:03.641567 25537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:03.641796 25537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:03.641820 25537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:03.642992 25537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:03.643033 25537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:03.643047 25537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:03.643060 25537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:03.643072 25537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:03.644722 25537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:46019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:03.644994 25537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:03.645308 25537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:03.651247 25542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:03.651297 25545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:03.651433 25537 server_base.cc:1034] running on GCE node
W20260926 09:22:03.651247 25543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:03.652014 25537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:03.652333 25537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:03.653476 25537 hybrid_clock.cc:648] HybridClock initialized: now 1790414523653464 us; error 33 us; skew 500 ppm
Sep 26 09:22:03 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414523, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:03.656273 25537 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:03.656986 25537 webserver.cc:460] Webserver started at http://127.24.235.126:42455/ using document root <none> and password file <none>
I20260926 09:22:03.657375 25537 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:03.657429 25537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:03.657617 25537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:03.658876 25537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "938b6fd93cf7460691141c6459826312"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "77068ca472e938ba7d750d46133e611c"
server_key_iv: "c072c355de66312a296ec682ec0c8610"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.659323 25537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "938b6fd93cf7460691141c6459826312"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "77068ca472e938ba7d750d46133e611c"
server_key_iv: "c072c355de66312a296ec682ec0c8610"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.661221 25537 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 09:22:03.662191 25552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:03.662438 25537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:03.662557 25537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "938b6fd93cf7460691141c6459826312"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "77068ca472e938ba7d750d46133e611c"
server_key_iv: "c072c355de66312a296ec682ec0c8610"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.662746 25537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:03.676156 25537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:03.678879 25537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:03.679070 25537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:03.684729 25537 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:46019
I20260926 09:22:03.684757 25604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:46019 every 8 connection(s)
I20260926 09:22:03.685341 25537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:03.686265 25605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:03.688812 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312: Bootstrap starting.
I20260926 09:22:03.689544 25605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:03.689627 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 25537
I20260926 09:22:03.689756 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:03.689998 25605 log.cc:824] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:03.690543 25517 external_mini_cluster.cc:1428] Setting key 5d2ca68e58c31290575f276c39144b36
I20260926 09:22:03.690871 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312: No bootstrap required, opened a new log
I20260926 09:22:03.692940 25605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } }
I20260926 09:22:03.693161 25605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:03.693216 25605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938b6fd93cf7460691141c6459826312, State: Initialized, Role: FOLLOWER
I20260926 09:22:03.693379 25605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } }
I20260926 09:22:03.693480 25605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:03.693523 25605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:03.693571 25605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:03.694676 25605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } }
I20260926 09:22:03.694876 25605 leader_election.cc:302] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938b6fd93cf7460691141c6459826312; no voters:
I20260926 09:22:03.695062 25605 leader_election.cc:288] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:03.695109 25610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:03.695283 25610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [term 1 LEADER]: Becoming Leader. State: Replica: 938b6fd93cf7460691141c6459826312, State: Running, Role: LEADER
I20260926 09:22:03.695371 25605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:03.695472 25610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } }
I20260926 09:22:03.695936 25610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938b6fd93cf7460691141c6459826312. Latest consensus state: current_term: 1 leader_uuid: "938b6fd93cf7460691141c6459826312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } } }
I20260926 09:22:03.696190 25610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:03.696121 25612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938b6fd93cf7460691141c6459826312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938b6fd93cf7460691141c6459826312" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46019 } } }
I20260926 09:22:03.696434 25612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:03.697158 25619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:03.697935 25619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:03.699853 25619 catalog_manager.cc:1347] Generated new cluster ID: 9757888a35bf48b787fd9d46a17ed8da
I20260926 09:22:03.699914 25619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:03 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:03.708693 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:03.691582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.24.235.126:46019)
0926 09:22:03.691768 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:03.691778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:03.692043 (+ 265us) server_negotiation.cc:407] Connection header received
0926 09:22:03.694075 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:03.694084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:03.694148 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:03.694356 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:03.696613 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.698588 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.699836 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.700336 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.700475 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:03.704728 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:03.704732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:03.704733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:03.704744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:03.706586 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:03.707042 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:03.707046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:03.707047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:03.707165 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:03.707448 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:03.707449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:03.707450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:03.707694 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:03.707902 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:03.708144 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 09:22:03.708300 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260926 09:22:03.710865 25619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:03.711580 25619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:03.724107 25619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938b6fd93cf7460691141c6459826312: Generated new TSK 0
I20260926 09:22:03.724321 25619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:03.760305 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:03.873941 25631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:03.874204 25631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:03.874233 25631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:03.876111 25631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:03.876181 25631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:03.876219 25631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:03.878283 25631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:03.878602 25631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:03.878907 25631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:03.887377 25637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:03.887580 25631 server_base.cc:1034] running on GCE node
W20260926 09:22:03.887295 25636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:03.887503 25639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:03.887966 25631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:03.888338 25631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:03.889542 25631 hybrid_clock.cc:648] HybridClock initialized: now 1790414523889511 us; error 47 us; skew 500 ppm
Sep 26 09:22:03 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414523, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:03.893321 25631 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:03.893870 25631 webserver.cc:460] Webserver started at http://127.24.235.65:33137/ using document root <none> and password file <none>
I20260926 09:22:03.894109 25631 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:03.894155 25631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:03.894299 25631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:03.895447 25631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "af7faf4be18f4c7a90498d1df0bfda26"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "1050e3a31dc159d05c37c074db52d965"
server_key_iv: "5ffd918d402fb24fdcafd290da2b4ae8"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.895905 25631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "af7faf4be18f4c7a90498d1df0bfda26"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "1050e3a31dc159d05c37c074db52d965"
server_key_iv: "5ffd918d402fb24fdcafd290da2b4ae8"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.897707 25631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 09:22:03.898562 25646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:03.898716 25631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:03.898828 25631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "af7faf4be18f4c7a90498d1df0bfda26"
format_stamp: "Formatted at 2026-09-26 09:22:03 on dist-test-slave-7zlw"
server_key: "1050e3a31dc159d05c37c074db52d965"
server_key_iv: "5ffd918d402fb24fdcafd290da2b4ae8"
server_key_version: "encryptionkey@0"
I20260926 09:22:03.898909 25631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:03.917392 25631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:03.919829 25631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:03.919962 25631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:03.920291 25631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:03.920665 25631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:03.920781 25631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:03.920828 25631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:03.920854 25631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:03.932116 25631 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46469
I20260926 09:22:03.932251 25759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46469 every 8 connection(s)
I20260926 09:22:03.932564 25631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:03.936898 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 25631
I20260926 09:22:03.937050 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:03.937268 25517 external_mini_cluster.cc:1428] Setting key 3a7ac98937eb73fa761dea5ef178f34f
Sep 26 09:22:03 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414523, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:03.944033 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:03.933773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:47327 (local address 127.24.235.126:46019)
0926 09:22:03.933862 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:03.933865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:03.933990 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:22:03.934872 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:03.934874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:03.934892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:03.934945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:03.936306 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.936822 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.938278 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.938475 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.938562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:03.941176 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:03.941178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:03.941178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:03.941186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:03.942604 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:03.943035 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:03.943037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:03.943037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:03.943093 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:03.943347 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:03.943348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:03.943349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:03.943482 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:03.943564 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:03.943845 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 09:22:03.943936 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 09:22:03.944330 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:03.933489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46019 (local address 127.24.235.65:47327)
0926 09:22:03.933782 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:03.933867 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:03.934749 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:03.935033 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:03.935037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:03.935869 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:03.936198 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:03.936203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.936927 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.936928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:03.938185 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:03.938188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:03.938586 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:03.938587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:03.938744 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:03.939452 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:03.939461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:03.941085 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:03.942705 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:03.942713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:03.942715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:03.942950 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:03.943158 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:03.943160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:03.943160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:03.943277 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:03.943556 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:03.943562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:03.943711 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 09:22:03.943823 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 09:22:03.943994 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260926 09:22:03.945147 25760 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46019
I20260926 09:22:03.945490 25760 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:03.945770 25760 heartbeater.cc:511] Master 127.24.235.126:46019 requested a full tablet report, sending...
I20260926 09:22:03.946419 25569 ts_manager.cc:194] Registered new tserver with Master: af7faf4be18f4c7a90498d1df0bfda26 (127.24.235.65:46469)
I20260926 09:22:03.947177 25569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:47327
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:03.969626 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:04.087865 25765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:04.088114 25765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:04.088145 25765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:04.089601 25765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:04.089658 25765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:04.089691 25765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:04.091686 25765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:04.091987 25765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:04.092254 25765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:04.098636 25773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:04.098788 25771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:04.098858 25770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:04.098912 25765 server_base.cc:1034] running on GCE node
I20260926 09:22:04.099148 25765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:04.099421 25765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:04.100600 25765 hybrid_clock.cc:648] HybridClock initialized: now 1790414524100585 us; error 51 us; skew 500 ppm
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:04.103801 25765 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:04.104384 25765 webserver.cc:460] Webserver started at http://127.24.235.66:35413/ using document root <none> and password file <none>
I20260926 09:22:04.104635 25765 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:04.104703 25765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:04.104856 25765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:04.105973 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "ce34f45e62f545f9809ac8777df631e3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "6ceab99bcd2ded7a4f04b3952df241d7"
server_key_iv: "65b724e893f7157147a5c26e2a930766"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.106383 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "ce34f45e62f545f9809ac8777df631e3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "6ceab99bcd2ded7a4f04b3952df241d7"
server_key_iv: "65b724e893f7157147a5c26e2a930766"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.107929 25765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:04.108855 25780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.109107 25765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:04.109203 25765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "ce34f45e62f545f9809ac8777df631e3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "6ceab99bcd2ded7a4f04b3952df241d7"
server_key_iv: "65b724e893f7157147a5c26e2a930766"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.109311 25765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:04.123016 25765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:04.125176 25765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:04.125419 25765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:04.125809 25765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:04.126180 25765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:04.126235 25765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.126276 25765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:04.126298 25765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.137317 25765 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33309
I20260926 09:22:04.137362 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33309 every 8 connection(s)
I20260926 09:22:04.137704 25765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:04.146081 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 25765
I20260926 09:22:04.146184 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:04.146358 25517 external_mini_cluster.cc:1428] Setting key 46c093b1e707c750652e99bf07d86bfd
I20260926 09:22:04.147944 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.138418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:46567 (local address 127.24.235.126:46019)
0926 09:22:04.138466 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:04.138467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:04.138778 (+ 311us) server_negotiation.cc:407] Connection header received
0926 09:22:04.139311 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:04.139312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:04.139335 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:04.139401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:04.140356 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.140865 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.141871 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.142114 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.142183 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.144589 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:04.144592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:04.144592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:04.144606 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:04.146271 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.146818 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.146819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.146820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.146888 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.147195 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.147197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.147197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.147327 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:04.147407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:04.147744 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 09:22:04.147846 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:22:04.148195 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.138397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46019 (local address 127.24.235.66:46567)
0926 09:22:04.138581 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:04.138628 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:04.139220 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:04.139474 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:04.139478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:04.140047 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:04.140265 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:04.140270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.140947 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.140948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:04.141769 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:04.141771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.142207 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.142208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:04.142317 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.142822 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:04.142830 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:04.144405 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:04.146363 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:04.146370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:04.146373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:04.146725 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:04.146960 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:04.146960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:04.146969 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:04.147131 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:04.147412 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:04.147419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:04.147598 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 09:22:04.147721 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 09:22:04.147900 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260926 09:22:04.148932 25894 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46019
I20260926 09:22:04.149088 25894 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:04.149317 25894 heartbeater.cc:511] Master 127.24.235.126:46019 requested a full tablet report, sending...
I20260926 09:22:04.149757 25569 ts_manager.cc:194] Registered new tserver with Master: ce34f45e62f545f9809ac8777df631e3 (127.24.235.66:33309)
I20260926 09:22:04.150141 25569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:46567
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:04.177887 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:04.285147 25899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:04.285356 25899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:04.285382 25899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:04.286566 25899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:04.286612 25899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:04.286638 25899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:04.288082 25899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:04.288352 25899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:04.288597 25899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:04.294063 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:04.294210 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:04.294374 25899 server_base.cc:1034] running on GCE node
W20260926 09:22:04.294149 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:04.294621 25899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:04.294910 25899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:04.296068 25899 hybrid_clock.cc:648] HybridClock initialized: now 1790414524296068 us; error 42 us; skew 500 ppm
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:04.299495 25899 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:04.300029 25899 webserver.cc:460] Webserver started at http://127.24.235.67:34035/ using document root <none> and password file <none>
I20260926 09:22:04.300284 25899 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:04.300355 25899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:04.300539 25899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:04.301930 25899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "f87b70beb7724d3ea810076ad9df8cc3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "7893e33536112fef62940fc17a93e93e"
server_key_iv: "40904b9e0e4f4d9800bf594b7a5f1c87"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.302383 25899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "f87b70beb7724d3ea810076ad9df8cc3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "7893e33536112fef62940fc17a93e93e"
server_key_iv: "40904b9e0e4f4d9800bf594b7a5f1c87"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.304395 25899 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 09:22:04.305439 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.305653 25899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:04.305766 25899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "f87b70beb7724d3ea810076ad9df8cc3"
format_stamp: "Formatted at 2026-09-26 09:22:04 on dist-test-slave-7zlw"
server_key: "7893e33536112fef62940fc17a93e93e"
server_key_iv: "40904b9e0e4f4d9800bf594b7a5f1c87"
server_key_version: "encryptionkey@0"
I20260926 09:22:04.305845 25899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:04.323460 25899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:04.326248 25899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:04.326481 25899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:04.326912 25899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:04.327292 25899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:04.327354 25899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.327402 25899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:04.327451 25899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:04.340844 25899 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:37943
I20260926 09:22:04.341044 26027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:37943 every 8 connection(s)
I20260926 09:22:04.341358 25899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:04.344393 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 25899
I20260926 09:22:04.344476 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:04.344664 25517 external_mini_cluster.cc:1428] Setting key 52b9c91f1c3b05c548be25eb50b9c314
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:04.353601 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.342292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:38653 (local address 127.24.235.126:46019)
0926 09:22:04.342365 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:04.342367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:04.342681 (+ 314us) server_negotiation.cc:407] Connection header received
0926 09:22:04.343476 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:04.343477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:04.343491 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:04.343590 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:04.344852 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.345586 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.346843 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.347051 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.347141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.350101 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:04.350103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:04.350103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:04.350110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:04.351931 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.352399 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.352400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.352401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.352468 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.352795 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.352796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.352797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.352927 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:04.352992 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:04.353379 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 09:22:04.353528 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:22:04.353847 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.342263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46019 (local address 127.24.235.67:38653)
0926 09:22:04.342501 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:04.342542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:04.343365 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:04.343695 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:04.343699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:04.344394 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:04.344712 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:04.344718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.345671 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.345672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:04.346745 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:04.346748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.347149 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.347150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:04.347304 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.348016 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:04.348025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:04.349986 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:04.352027 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:04.352033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:04.352035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:04.352291 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:04.352580 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:04.352581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:04.352582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:04.352710 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:04.353065 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:04.353072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:04.353221 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 09:22:04.353327 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:22:04.353494 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260926 09:22:04.354681 26028 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46019
I20260926 09:22:04.354830 26028 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:04.355046 26028 heartbeater.cc:511] Master 127.24.235.126:46019 requested a full tablet report, sending...
I20260926 09:22:04.355512 25569 ts_manager.cc:194] Registered new tserver with Master: f87b70beb7724d3ea810076ad9df8cc3 (127.24.235.67:37943)
I20260926 09:22:04.355959 25569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:38653
I20260926 09:22:04.356840 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:04.384884 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.375495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.24.235.65:46469)
0926 09:22:04.375726 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:04.375732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:04.375813 (+ 81us) server_negotiation.cc:407] Connection header received
0926 09:22:04.375907 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:04.375910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:04.375924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:04.376010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:04.376696 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.377643 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.378413 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.378858 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.379052 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.381784 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:04.381792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:04.381792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:04.381800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:04.383650 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.384008 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.384011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.384011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.384071 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.384298 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.384299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.384299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.384436 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:04.384653 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:04.384690 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 09:22:04.384755 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
W20260926 09:22:04.385396 25674 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43728
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[25524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:04.410656 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:04.402488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.24.235.65:46469)
0926 09:22:04.402550 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:04.402552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:04.402743 (+ 191us) server_negotiation.cc:407] Connection header received
0926 09:22:04.402827 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:04.402828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:04.402844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:04.402907 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:04.403521 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.404050 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.404791 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:04.404998 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:04.405146 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:04.407490 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:04.407492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:04.407493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:04.407506 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:04.409269 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.409754 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.409755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.409756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.409823 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:04.410038 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:04.410039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:04.410040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:04.410170 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:04.410346 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:04.410414 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 09:22:04.410515 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:22:04.411298 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 25631
I20260926 09:22:04.416548 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 25765
I20260926 09:22:04.421355 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 25899
I20260926 09:22:04.426055 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 25537
2026-09-26T09:22:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2160 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[26042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[26042](info): set up 2 sockets
Sep 26 09:22:04 dist-test-slave-7zlw krb5kdc[26042](info): commencing operation
krb5kdc: starting...
W20260926 09:22:05.528223 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 09:22:05.710060 25517 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.182s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 09:22:05.823684 25517 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.113s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 09:22:06.030874 25517 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.207s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/test-user.keytab.
W20260926 09:22:06.562459 25517 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.531s user 0.000s sys 0.002s
Sep 26 09:22:06 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:06.656203 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46433
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:46433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:06.741816 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:06.742071 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:06.742108 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:06.743287 26055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:06.743358 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:06.743384 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:06.743414 26055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:06.743438 26055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:06.744962 26055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:46433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:06.745263 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:06.745520 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:06.750553 26055 server_base.cc:1034] running on GCE node
W20260926 09:22:06.750459 26060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:06.750470 26061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:06.750459 26063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:06.750995 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:06.751302 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:06.752491 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790414526752465 us; error 46 us; skew 500 ppm
Sep 26 09:22:06 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:06.755050 26055 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:06.755470 26055 webserver.cc:460] Webserver started at http://127.24.235.126:33705/ using document root <none> and password file <none>
I20260926 09:22:06.755713 26055 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:06.755776 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:06.755919 26055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:06.757143 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "b5f99538ffc341a8b60e69cddd912637"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "fba951dd39c540899eadbd7a9221fb7b"
server_key_iv: "eea4e3dcf6a0a8b256f45068b0602621"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.757493 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "b5f99538ffc341a8b60e69cddd912637"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "fba951dd39c540899eadbd7a9221fb7b"
server_key_iv: "eea4e3dcf6a0a8b256f45068b0602621"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.758797 26055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:06.759550 26070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:06.759718 26055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:06.759829 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "b5f99538ffc341a8b60e69cddd912637"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "fba951dd39c540899eadbd7a9221fb7b"
server_key_iv: "eea4e3dcf6a0a8b256f45068b0602621"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.759907 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:06.786396 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:06.788961 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:06.789192 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:06.793192 26055 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:46433
I20260926 09:22:06.793272 26122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:46433 every 8 connection(s)
I20260926 09:22:06.793653 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:06.794595 26123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:06.796900 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637: Bootstrap starting.
I20260926 09:22:06.797530 26123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:06.797816 26123 log.cc:824] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:06.798569 26123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637: No bootstrap required, opened a new log
I20260926 09:22:06.800115 26123 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } }
I20260926 09:22:06.800247 26123 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:06.800271 26123 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f99538ffc341a8b60e69cddd912637, State: Initialized, Role: FOLLOWER
I20260926 09:22:06.800438 26123 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } }
I20260926 09:22:06.800499 26123 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:06.800527 26123 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:06.800588 26123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:06.801160 26123 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } }
I20260926 09:22:06.801271 26123 leader_election.cc:302] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f99538ffc341a8b60e69cddd912637; no voters:
I20260926 09:22:06.801427 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26055
I20260926 09:22:06.801447 26123 leader_election.cc:288] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:06.801474 26126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:06.801520 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:06.801586 26126 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [term 1 LEADER]: Becoming Leader. State: Replica: b5f99538ffc341a8b60e69cddd912637, State: Running, Role: LEADER
I20260926 09:22:06.801658 26126 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } }
I20260926 09:22:06.801698 25517 external_mini_cluster.cc:1428] Setting key d1837bf713ef6aa3b4879750b80bd151
I20260926 09:22:06.801718 26123 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:06.801962 26126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f99538ffc341a8b60e69cddd912637. Latest consensus state: current_term: 1 leader_uuid: "b5f99538ffc341a8b60e69cddd912637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } } }
I20260926 09:22:06.802024 26126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:06.802223 26130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:06.802234 26127 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f99538ffc341a8b60e69cddd912637" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f99538ffc341a8b60e69cddd912637" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46433 } } }
I20260926 09:22:06.802424 26127 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:06.803021 26130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:06.804754 26130 catalog_manager.cc:1347] Generated new cluster ID: 93b13713941a48f7a4a5d06fffc1c4a5
I20260926 09:22:06.804805 26130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:06 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:06.811691 26130 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:06.812461 26130 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:06.812695 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:06.802591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.24.235.126:46433)
0926 09:22:06.802905 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:06.802915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:06.802948 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:22:06.803072 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:06.803081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:06.803142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:06.803309 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:06.804153 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:06.805371 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:06.806155 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:06.806610 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:06.806756 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:06.809208 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:06.809212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:06.809212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:06.809225 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:06.810890 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:06.811267 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:06.811271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:06.811272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:06.811378 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:06.811590 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:06.811591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:06.811592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:06.811911 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:06.812105 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:06.812305 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 09:22:06.812458 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260926 09:22:06.826727 26130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f99538ffc341a8b60e69cddd912637: Generated new TSK 0
I20260926 09:22:06.826953 26130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:06.860482 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:06.952831 26149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:06.953120 26149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:06.953166 26149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:06.954356 26149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:06.954420 26149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:06.954473 26149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:06.956013 26149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:06.956275 26149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:06.956526 26149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:06.962018 26154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:06.962056 26155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:06.962122 26149 server_base.cc:1034] running on GCE node
W20260926 09:22:06.962072 26157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:06.962419 26149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:06.962707 26149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:06.963929 26149 hybrid_clock.cc:648] HybridClock initialized: now 1790414526963908 us; error 36 us; skew 500 ppm
Sep 26 09:22:06 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:06.966531 26149 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:06.966998 26149 webserver.cc:460] Webserver started at http://127.24.235.65:35973/ using document root <none> and password file <none>
I20260926 09:22:06.967231 26149 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:06.967300 26149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:06.967446 26149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:06.968645 26149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "4cca8791f04f453787fe03366800d2bf"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "9cf96b06f91cfd18bf7f718ba988e5b9"
server_key_iv: "f74141ecd800e65956309bd5606a5020"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.969089 26149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "4cca8791f04f453787fe03366800d2bf"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "9cf96b06f91cfd18bf7f718ba988e5b9"
server_key_iv: "f74141ecd800e65956309bd5606a5020"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.970947 26149 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:06.971908 26164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:06.972100 26149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:06.972199 26149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "4cca8791f04f453787fe03366800d2bf"
format_stamp: "Formatted at 2026-09-26 09:22:06 on dist-test-slave-7zlw"
server_key: "9cf96b06f91cfd18bf7f718ba988e5b9"
server_key_iv: "f74141ecd800e65956309bd5606a5020"
server_key_version: "encryptionkey@0"
I20260926 09:22:06.972285 26149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:06.987732 26149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:06.990072 26149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:06.990245 26149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:06.990490 26149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:06.990831 26149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:06.990887 26149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:06.990921 26149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:06.990948 26149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:06.996881 26149 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:40309
I20260926 09:22:06.996953 26277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:40309 every 8 connection(s)
I20260926 09:22:06.997380 26149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:07.006845 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26149
I20260926 09:22:07.006974 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:07.007161 25517 external_mini_cluster.cc:1428] Setting key b6d3412cd336d73295555ba183a2cf93
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:07.012734 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:06.998535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:58181 (local address 127.24.235.126:46433)
0926 09:22:06.998608 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:06.998611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.001064 (+ 2453us) server_negotiation.cc:407] Connection header received
0926 09:22:07.002556 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.002558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.002575 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.002663 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.003601 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.004385 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.006049 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.006467 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.006543 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.009472 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.009474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.009475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.009483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.011042 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.011608 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.011610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.011610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.011672 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.012011 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.012012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.012012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.012137 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.012214 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.012519 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.012637 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:07.012913 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:06.998511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46433 (local address 127.24.235.65:58181)
0926 09:22:07.000880 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.000923 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.002418 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.002757 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.002761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.003157 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:07.003483 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.003489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.004525 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.004526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.005942 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.005945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.006625 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.006626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.006781 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.007473 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:07.007483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:07.009372 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:07.011128 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.011134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.011136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.011489 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.011761 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.011768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.011769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.011899 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.012211 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:07.012219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:07.012371 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.012484 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.012663 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2300,"thread_start_us":110,"threads_started":1}
I20260926 09:22:07.013463 26278 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46433
I20260926 09:22:07.013547 26278 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:07.013744 26278 heartbeater.cc:511] Master 127.24.235.126:46433 requested a full tablet report, sending...
I20260926 09:22:07.014271 26087 ts_manager.cc:194] Registered new tserver with Master: 4cca8791f04f453787fe03366800d2bf (127.24.235.65:40309)
I20260926 09:22:07.014878 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:58181
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:07.036448 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:07.128118 26283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:07.128355 26283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:07.128381 26283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:07.129627 26283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:07.129673 26283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:07.129699 26283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:07.131152 26283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:07.131342 26283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:07.131529 26283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:07.136724 26289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:07.136718 26291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:07.136880 26283 server_base.cc:1034] running on GCE node
W20260926 09:22:07.136999 26288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:07.137347 26283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:07.137668 26283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:07.138859 26283 hybrid_clock.cc:648] HybridClock initialized: now 1790414527138844 us; error 42 us; skew 500 ppm
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:07.141883 26283 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:07.142405 26283 webserver.cc:460] Webserver started at http://127.24.235.66:32965/ using document root <none> and password file <none>
I20260926 09:22:07.142645 26283 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:07.142712 26283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:07.142860 26283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:07.143934 26283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "4cce2703f63243bc96621d4ff7fbda8a"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "69d3f2b2c989a17a1a946558d2c9484a"
server_key_iv: "cccd42f3db6388dd042d5a72563a90f6"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.144318 26283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "4cce2703f63243bc96621d4ff7fbda8a"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "69d3f2b2c989a17a1a946558d2c9484a"
server_key_iv: "cccd42f3db6388dd042d5a72563a90f6"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.145795 26283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:07.146611 26298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.146785 26283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:07.146889 26283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "4cce2703f63243bc96621d4ff7fbda8a"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "69d3f2b2c989a17a1a946558d2c9484a"
server_key_iv: "cccd42f3db6388dd042d5a72563a90f6"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.146966 26283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:07.165110 26283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:07.167555 26283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:07.167721 26283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:07.167958 26283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:07.168293 26283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:07.168354 26283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.168391 26283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:07.168418 26283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.174438 26283 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:37001
I20260926 09:22:07.174482 26411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:37001 every 8 connection(s)
I20260926 09:22:07.174840 26283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:07.182118 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26283
I20260926 09:22:07.182199 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:07.182363 25517 external_mini_cluster.cc:1428] Setting key 43f9d898e3a38b5030be4f72f8e36260
I20260926 09:22:07.185364 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.175731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60909 (local address 127.24.235.126:46433)
0926 09:22:07.175783 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.175786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.175959 (+ 173us) server_negotiation.cc:407] Connection header received
0926 09:22:07.176953 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.176954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.176963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.177066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.177733 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.178271 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.179308 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.179506 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.179569 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.181981 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.181982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.181982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.181989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.183648 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.184194 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.184195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.184195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.184252 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.184582 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.184584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.184584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.184741 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.184829 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.185172 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.185269 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:22:07.185592 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.175676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46433 (local address 127.24.235.66:60909)
0926 09:22:07.175834 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.175876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.176858 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.177148 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.177152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.177412 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:07.177614 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.177618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.178369 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.178370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.179216 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.179218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.179613 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.179614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.179727 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.180378 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:07.180387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:07.181876 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:07.183761 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.183769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.183771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.184107 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.184312 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.184313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.184313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.184463 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.184839 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:07.184846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:07.184988 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.185152 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.185325 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":101,"threads_started":1}
I20260926 09:22:07.186221 26412 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46433
I20260926 09:22:07.186333 26412 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:07.186527 26412 heartbeater.cc:511] Master 127.24.235.126:46433 requested a full tablet report, sending...
I20260926 09:22:07.186923 26087 ts_manager.cc:194] Registered new tserver with Master: 4cce2703f63243bc96621d4ff7fbda8a (127.24.235.66:37001)
I20260926 09:22:07.187311 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:60909
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:07.211759 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:07.305744 26417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:07.305976 26417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:07.306005 26417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:07.307268 26417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:07.307309 26417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:07.307339 26417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:07.309322 26417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46433
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:07.309585 26417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:07.309787 26417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:07.315614 26423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:07.315636 26422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:07.315718 26425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:07.315852 26417 server_base.cc:1034] running on GCE node
I20260926 09:22:07.316077 26417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:07.316342 26417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:07.317525 26417 hybrid_clock.cc:648] HybridClock initialized: now 1790414527317493 us; error 56 us; skew 500 ppm
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:07.320859 26417 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:07.321411 26417 webserver.cc:460] Webserver started at http://127.24.235.67:43115/ using document root <none> and password file <none>
I20260926 09:22:07.321642 26417 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:07.321709 26417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:07.321863 26417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:07.323187 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "2f146f265e00a115ac8bbf3d999165b4"
server_key_iv: "06dd3b3d8cb361f11188be6613e32418"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.323611 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "2f146f265e00a115ac8bbf3d999165b4"
server_key_iv: "06dd3b3d8cb361f11188be6613e32418"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.325182 26417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:07.326162 26432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.326400 26417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:07.326516 26417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4"
format_stamp: "Formatted at 2026-09-26 09:22:07 on dist-test-slave-7zlw"
server_key: "2f146f265e00a115ac8bbf3d999165b4"
server_key_iv: "06dd3b3d8cb361f11188be6613e32418"
server_key_version: "encryptionkey@0"
I20260926 09:22:07.326623 26417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:07.335124 26417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:07.337750 26417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:07.337883 26417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:07.338114 26417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:07.338505 26417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:07.338543 26417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.338568 26417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:07.338581 26417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:07.348067 26417 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:40845
I20260926 09:22:07.348250 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:40845 every 8 connection(s)
I20260926 09:22:07.348543 26417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:07.357239 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26417
I20260926 09:22:07.357344 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:07.357513 25517 external_mini_cluster.cc:1428] Setting key 053e450c742a8b3f86a19517b3bb4f9e
I20260926 09:22:07.360585 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.349702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:40323 (local address 127.24.235.126:46433)
0926 09:22:07.349784 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.349787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.350214 (+ 427us) server_negotiation.cc:407] Connection header received
0926 09:22:07.351932 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.351934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.351951 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.352018 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.352803 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.353391 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.354517 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.354726 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.354833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.357397 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.357400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.357400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.357410 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.359159 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.359605 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.359607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.359607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.359665 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.359913 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.359914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.359915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.360051 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.360121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.360402 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.360484 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 09:22:07.360859 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.349692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46433 (local address 127.24.235.67:40323)
0926 09:22:07.350067 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.350106 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.351793 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.352097 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.352100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.352416 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:07.352690 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.352696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.353506 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.353508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.354399 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.354402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.354842 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.354843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.354984 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.355526 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:07.355536 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:07.357276 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:07.359262 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.359269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.359272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.359516 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.359729 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:07.359730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:07.359730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:07.359846 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:07.360122 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:07.360129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:07.360273 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.360382 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.360561 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":208,"threads_started":1}
I20260926 09:22:07.361392 26546 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46433
I20260926 09:22:07.361512 26546 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:07.361720 26546 heartbeater.cc:511] Master 127.24.235.126:46433 requested a full tablet report, sending...
I20260926 09:22:07.362222 26087 ts_manager.cc:194] Registered new tserver with Master: 8029c1467d3d4463bb7e4bfce3a2eeb4 (127.24.235.67:40845)
I20260926 09:22:07.362689 26087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:40323
I20260926 09:22:07.369661 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:07.378422 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.371414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.24.235.126:46433)
0926 09:22:07.371609 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.371612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.371699 (+ 87us) server_negotiation.cc:407] Connection header received
0926 09:22:07.371796 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.371797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.371809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.371874 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.372580 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.373197 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.373975 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.374198 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.374289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.375709 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.375711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.375712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.375721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.377104 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.377469 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.377471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.377471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.377550 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.377761 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.377762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.377762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.377935 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.378057 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.378206 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.378304 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 09:22:07.380976 26087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:07.382939 26087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:07.390499 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.385802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40309 (local address 127.0.0.1:56678)
0926 09:22:07.386281 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.386331 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.386416 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.386719 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.386723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.386725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:07.386810 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.386813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.387997 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.387998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.389260 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.389262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390116 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.390116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.390219 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.390249 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.390355 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.390421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":61,"threads_started":1}
I20260926 09:22:07.391088 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.385819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:40845 (local address 127.0.0.1:38462)
0926 09:22:07.386525 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.386534 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.386608 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.386998 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.386999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.387002 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:07.387112 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.387115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.388388 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.388389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.389852 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.389855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390848 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.390849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.390910 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.390913 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.390968 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.391004 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"spinlock_wait_cycles":768,"thread_start_us":254,"threads_started":1}
I20260926 09:22:07.391268 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.386113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.24.235.65:40309)
0926 09:22:07.386489 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.386492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.386503 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:07.386560 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.386562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.386576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.386688 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:07.386885 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.387846 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.389361 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390028 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.390107 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.390177 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.391082 (+ 905us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.391142 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":166,"threads_started":1}
I20260926 09:22:07.391132 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.386025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.24.235.67:40845)
0926 09:22:07.386279 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.386285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.386625 (+ 340us) server_negotiation.cc:407] Connection header received
0926 09:22:07.386685 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.386688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.386706 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.386835 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:07.387223 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.388155 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.389950 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390685 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.390758 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.390810 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.390988 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.391038 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
I20260926 09:22:07.392207 26480 tablet_service.cc:1467] Processing CreateTablet for tablet 2d270fe5396a47d5a49f99bec55c2726 (DEFAULT_TABLE table=test-table [id=a7608664dc23475fbec0b79a129149aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:07.392148 26212 tablet_service.cc:1467] Processing CreateTablet for tablet 2d270fe5396a47d5a49f99bec55c2726 (DEFAULT_TABLE table=test-table [id=a7608664dc23475fbec0b79a129149aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:07.392604 26212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d270fe5396a47d5a49f99bec55c2726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:07.392599 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d270fe5396a47d5a49f99bec55c2726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:07.392781 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.385819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37001 (local address 127.0.0.1:44728)
0926 09:22:07.388062 (+ 2243us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.388073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.388147 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.388643 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.388644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.388646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:07.388684 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.388685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390383 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.390384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.391865 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.391867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.392519 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.392520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.392584 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.392588 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.392670 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.392710 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"mutex_wait_us":2,"thread_start_us":620,"threads_started":1}
I20260926 09:22:07.392810 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.386137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.24.235.66:37001)
0926 09:22:07.386457 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.386461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.388146 (+ 1685us) server_negotiation.cc:407] Connection header received
0926 09:22:07.388288 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.388291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.388312 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.388409 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:07.388774 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.390152 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.391952 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.392431 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.392515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.392567 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.392694 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.392734 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":178,"threads_started":1}
I20260926 09:22:07.393739 26346 tablet_service.cc:1467] Processing CreateTablet for tablet 2d270fe5396a47d5a49f99bec55c2726 (DEFAULT_TABLE table=test-table [id=a7608664dc23475fbec0b79a129149aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:07.394106 26346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d270fe5396a47d5a49f99bec55c2726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:07.396708 26563 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Bootstrap starting.
I20260926 09:22:07.397179 26564 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Bootstrap starting.
I20260926 09:22:07.397535 26563 tablet_bootstrap.cc:654] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:07.397809 26564 tablet_bootstrap.cc:654] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:07.398365 26563 log.cc:824] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:07.398438 26564 log.cc:824] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:07.398805 26565 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: Bootstrap starting.
I20260926 09:22:07.399345 26564 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: No bootstrap required, opened a new log
I20260926 09:22:07.399422 26564 ts_tablet_manager.cc:1397] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:07.399437 26565 tablet_bootstrap.cc:654] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:07.399514 26563 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: No bootstrap required, opened a new log
I20260926 09:22:07.399580 26563 ts_tablet_manager.cc:1397] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:22:07.399914 26565 log.cc:824] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:07.400735 26565 tablet_bootstrap.cc:492] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: No bootstrap required, opened a new log
I20260926 09:22:07.400794 26565 ts_tablet_manager.cc:1397] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:07.401684 26564 raft_consensus.cc:348] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.401698 26563 raft_consensus.cc:348] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.401890 26564 raft_consensus.cc:374] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:07.401890 26563 raft_consensus.cc:374] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:07.401949 26563 raft_consensus.cc:729] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cca8791f04f453787fe03366800d2bf, State: Initialized, Role: FOLLOWER
I20260926 09:22:07.401948 26564 raft_consensus.cc:729] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cce2703f63243bc96621d4ff7fbda8a, State: Initialized, Role: FOLLOWER
I20260926 09:22:07.402100 26563 consensus_queue.cc:260] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.402117 26564 consensus_queue.cc:260] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.402480 26564 ts_tablet_manager.cc:1428] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:07.402479 26563 ts_tablet_manager.cc:1428] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:07.402745 26278 heartbeater.cc:503] Master 127.24.235.126:46433 was elected leader, sending a full tablet report...
I20260926 09:22:07.402765 26565 raft_consensus.cc:348] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.402938 26565 raft_consensus.cc:374] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:07.402971 26565 raft_consensus.cc:729] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8029c1467d3d4463bb7e4bfce3a2eeb4, State: Initialized, Role: FOLLOWER
I20260926 09:22:07.403097 26565 consensus_queue.cc:260] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.403766 26565 ts_tablet_manager.cc:1428] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:22:07.403790 26546 heartbeater.cc:503] Master 127.24.235.126:46433 was elected leader, sending a full tablet report...
I20260926 09:22:07.405097 26412 heartbeater.cc:503] Master 127.24.235.126:46433 was elected leader, sending a full tablet report...
W20260926 09:22:07.425442 26413 tablet.cc:2369] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:07.446599 26571 raft_consensus.cc:482] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:07.446722 26571 raft_consensus.cc:504] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.447216 26571 leader_election.cc:288] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cca8791f04f453787fe03366800d2bf (127.24.235.65:40309), 4cce2703f63243bc96621d4ff7fbda8a (127.24.235.66:37001)
I20260926 09:22:07.450354 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.447552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:50767 (local address 127.24.235.66:37001)
0926 09:22:07.447602 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.447604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.447614 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:07.447643 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.447644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.447654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.447721 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:07.447927 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.448416 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.449539 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.449994 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.450091 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.450100 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.450231 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.450281 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:07.450366 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.447376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37001 (local address 127.24.235.67:50767)
0926 09:22:07.447433 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.447446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.447496 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.447754 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.447756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.447758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:07.447837 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.447839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.448519 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.448521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.449399 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.449401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.450071 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.450072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.450147 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.450151 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.450214 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.450275 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:22:07.450403 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.447499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:38845 (local address 127.24.235.65:40309)
0926 09:22:07.447542 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.447545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.447739 (+ 194us) server_negotiation.cc:407] Connection header received
0926 09:22:07.447768 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.447769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.447780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.447839 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:07.448079 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.448631 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.449645 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.450028 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.450111 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.450119 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.450289 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.450330 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:07.450398 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.447394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40309 (local address 127.24.235.67:38845)
0926 09:22:07.447586 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:07.447596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:07.447666 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:07.447895 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:07.447896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:07.447899 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:07.447989 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.447991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.448728 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.448730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.449555 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:07.449558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.450110 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.450112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:07.450212 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.450216 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:07.450272 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:22:07.450307 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":12,"thread_start_us":113,"threads_started":1}
I20260926 09:22:07.450835 26366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d270fe5396a47d5a49f99bec55c2726" candidate_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cce2703f63243bc96621d4ff7fbda8a" is_pre_election: true
I20260926 09:22:07.450838 26232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d270fe5396a47d5a49f99bec55c2726" candidate_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cca8791f04f453787fe03366800d2bf" is_pre_election: true
I20260926 09:22:07.451053 26232 raft_consensus.cc:2454] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8029c1467d3d4463bb7e4bfce3a2eeb4 in term 0.
I20260926 09:22:07.451054 26366 raft_consensus.cc:2454] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8029c1467d3d4463bb7e4bfce3a2eeb4 in term 0.
I20260926 09:22:07.451287 26433 leader_election.cc:302] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cca8791f04f453787fe03366800d2bf, 8029c1467d3d4463bb7e4bfce3a2eeb4; no voters:
I20260926 09:22:07.451408 26571 raft_consensus.cc:2781] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:07.451470 26571 raft_consensus.cc:482] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:07.451488 26571 raft_consensus.cc:3037] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:07.452162 26571 raft_consensus.cc:504] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.452301 26571 leader_election.cc:288] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [CANDIDATE]: Term 1 election: Requested vote from peers 4cca8791f04f453787fe03366800d2bf (127.24.235.65:40309), 4cce2703f63243bc96621d4ff7fbda8a (127.24.235.66:37001)
I20260926 09:22:07.452443 26232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d270fe5396a47d5a49f99bec55c2726" candidate_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cca8791f04f453787fe03366800d2bf"
I20260926 09:22:07.452451 26366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d270fe5396a47d5a49f99bec55c2726" candidate_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cce2703f63243bc96621d4ff7fbda8a"
I20260926 09:22:07.452531 26232 raft_consensus.cc:3037] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:07.452605 26366 raft_consensus.cc:3037] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:07.453488 26232 raft_consensus.cc:2454] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8029c1467d3d4463bb7e4bfce3a2eeb4 in term 1.
I20260926 09:22:07.453498 26366 raft_consensus.cc:2454] T 2d270fe5396a47d5a49f99bec55c2726 P 4cce2703f63243bc96621d4ff7fbda8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8029c1467d3d4463bb7e4bfce3a2eeb4 in term 1.
I20260926 09:22:07.453706 26436 leader_election.cc:302] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cce2703f63243bc96621d4ff7fbda8a, 8029c1467d3d4463bb7e4bfce3a2eeb4; no voters:
I20260926 09:22:07.453826 26571 raft_consensus.cc:2781] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:07.453999 26571 raft_consensus.cc:686] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [term 1 LEADER]: Becoming Leader. State: Replica: 8029c1467d3d4463bb7e4bfce3a2eeb4, State: Running, Role: LEADER
I20260926 09:22:07.454105 26571 consensus_queue.cc:237] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } }
I20260926 09:22:07.455101 26086 catalog_manager.cc:5495] T 2d270fe5396a47d5a49f99bec55c2726 P 8029c1467d3d4463bb7e4bfce3a2eeb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8029c1467d3d4463bb7e4bfce3a2eeb4 (127.24.235.67). New cstate: current_term: 1 leader_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cca8791f04f453787fe03366800d2bf" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cce2703f63243bc96621d4ff7fbda8a" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8029c1467d3d4463bb7e4bfce3a2eeb4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 40845 } health_report { overall_health: HEALTHY } } }
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:07.471211 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.462996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.24.235.65:40309)
0926 09:22:07.463087 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.463088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.463170 (+ 82us) server_negotiation.cc:407] Connection header received
0926 09:22:07.463198 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.463199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.463214 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.463266 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.463964 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.464439 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.465278 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.465476 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.465535 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.467982 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.467993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.467993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.468000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.469864 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.470303 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.470306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.470307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.470369 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.470645 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.470645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.470646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.470809 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.470953 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.471046 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.471123 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:07.495990 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.488397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.24.235.65:40309)
0926 09:22:07.488504 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.488506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.488608 (+ 102us) server_negotiation.cc:407] Connection header received
0926 09:22:07.488708 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.488709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.488719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.488790 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.489504 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.490028 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.490795 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.490994 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.491082 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.493391 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.493392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.493393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.493401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.494772 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.495126 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.495127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.495128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.495184 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.495378 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.495379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.495380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.495481 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.495596 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.495786 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.495869 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 09:22:07.498212 26279 tablet.cc:2369] T 2d270fe5396a47d5a49f99bec55c2726 P 4cca8791f04f453787fe03366800d2bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:07.520243 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:07.512696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.24.235.65:40309)
0926 09:22:07.512824 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:07.512827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:07.512931 (+ 104us) server_negotiation.cc:407] Connection header received
0926 09:22:07.513076 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:07.513077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:07.513089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:07.513155 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:07.513860 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.514341 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.515139 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:07.515442 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:07.515522 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:07.518040 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:07.518041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:07.518042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:07.518048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:07.519257 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.519580 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.519581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.519582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.519658 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:07.519828 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:07.519828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:07.519829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:07.519929 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:07.520022 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:07.520110 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:22:07.520174 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 09:22:07.520884 26192 tablet_service.cc:2908] Rejecting scan request for tablet 2d270fe5396a47d5a49f99bec55c2726: Uninitialized: clean time has not yet been initialized
I20260926 09:22:07.522423 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26149
I20260926 09:22:07.528000 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26283
I20260926 09:22:07.533241 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26417
I20260926 09:22:07.538084 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26055
2026-09-26T09:22:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3111 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26587](info): set up 2 sockets
Sep 26 09:22:07 dist-test-slave-7zlw krb5kdc[26587](info): commencing operation
krb5kdc: starting...
W20260926 09:22:08.585371 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:08 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:08.713161 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41391
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:08.795013 26600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:08.795241 26600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:08.795264 26600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:08.796382 26600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:08.796419 26600 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:22:08.796433 26600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:08.796446 26600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:08.796458 26600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:08.796470 26600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:08.798013 26600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41391
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:08.798226 26600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:08.798406 26600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:08.803490 26600 server_base.cc:1034] running on GCE node
W20260926 09:22:08.803459 26608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:08.803462 26605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:08.803459 26606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:08.803963 26600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:08.804243 26600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:08.805491 26600 hybrid_clock.cc:648] HybridClock initialized: now 1790414528805423 us; error 28 us; skew 500 ppm
Sep 26 09:22:08 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414528, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:08.808337 26600 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:08.808830 26600 webserver.cc:460] Webserver started at http://127.24.235.126:46215/ using document root <none> and password file <none>
I20260926 09:22:08.809124 26600 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:08.809190 26600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:08.809351 26600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:08.810420 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "f193c876954f470797ecde266f7a7ee5"
format_stamp: "Formatted at 2026-09-26 09:22:08 on dist-test-slave-7zlw"
server_key: "79f03cfb005772143c798fa1dcc1ec1b"
server_key_iv: "b5ce0f9779fe6531f937c57e31f65215"
server_key_version: "encryptionkey@0"
I20260926 09:22:08.810782 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "f193c876954f470797ecde266f7a7ee5"
format_stamp: "Formatted at 2026-09-26 09:22:08 on dist-test-slave-7zlw"
server_key: "79f03cfb005772143c798fa1dcc1ec1b"
server_key_iv: "b5ce0f9779fe6531f937c57e31f65215"
server_key_version: "encryptionkey@0"
I20260926 09:22:08.812093 26600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:08.812819 26615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:08.812984 26600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:08.813107 26600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "f193c876954f470797ecde266f7a7ee5"
format_stamp: "Formatted at 2026-09-26 09:22:08 on dist-test-slave-7zlw"
server_key: "79f03cfb005772143c798fa1dcc1ec1b"
server_key_iv: "b5ce0f9779fe6531f937c57e31f65215"
server_key_version: "encryptionkey@0"
I20260926 09:22:08.813192 26600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:08.830197 26600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:08.832345 26600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:08.832523 26600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:08.837642 26600 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41391
I20260926 09:22:08.837706 26677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41391 every 8 connection(s)
I20260926 09:22:08.838055 26600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:08.838258 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26600
I20260926 09:22:08.838370 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:08.838560 25517 external_mini_cluster.cc:1428] Setting key 53da16d12a7d583e1653a58bf6ebc631
I20260926 09:22:08.838845 26678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:08.842180 26678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Bootstrap starting.
I20260926 09:22:08.842900 26678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:08.843190 26678 log.cc:824] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:08.843924 26678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: No bootstrap required, opened a new log
Sep 26 09:22:08 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:08.846585 26678 raft_consensus.cc:348] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } }
I20260926 09:22:08.846784 26678 raft_consensus.cc:374] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:08.846837 26678 raft_consensus.cc:729] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f193c876954f470797ecde266f7a7ee5, State: Initialized, Role: FOLLOWER
I20260926 09:22:08.847011 26678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } }
I20260926 09:22:08.847108 26678 raft_consensus.cc:388] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:08.847185 26678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:08.847255 26678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:08.847820 26678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } }
I20260926 09:22:08.847949 26678 leader_election.cc:302] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f193c876954f470797ecde266f7a7ee5; no voters:
I20260926 09:22:08.848130 26678 leader_election.cc:288] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:08.848196 26683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:08.848348 26683 raft_consensus.cc:686] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [term 1 LEADER]: Becoming Leader. State: Replica: f193c876954f470797ecde266f7a7ee5, State: Running, Role: LEADER
I20260926 09:22:08.848433 26678 sys_catalog.cc:564] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:08.848505 26683 consensus_queue.cc:237] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } }
I20260926 09:22:08.848943 26683 sys_catalog.cc:455] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f193c876954f470797ecde266f7a7ee5. Latest consensus state: current_term: 1 leader_uuid: "f193c876954f470797ecde266f7a7ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } } }
I20260926 09:22:08.849057 26685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f193c876954f470797ecde266f7a7ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f193c876954f470797ecde266f7a7ee5" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41391 } } }
I20260926 09:22:08.849120 26683 sys_catalog.cc:458] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:08.849148 26685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:08.849341 26692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:08.849952 26692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:08.850773 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:08.839046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.24.235.126:41391)
0926 09:22:08.839222 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:08.839233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:08.839265 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:22:08.839963 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:08.839971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:08.840020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:08.840140 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:08.840941 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:08.841916 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:08.842983 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:08.843442 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:08.843569 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:08.846354 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:08.846358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:08.846358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:08.846369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:08.848469 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:08.848922 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:08.848924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:08.848924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:08.849054 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:08.849302 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:08.849303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:08.849303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:08.849634 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:08.850352 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:08.850432 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:22:08.850576 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260926 09:22:08.851727 26692 catalog_manager.cc:1347] Generated new cluster ID: c5be7bed696a4c679d7c03adf6d56537
I20260926 09:22:08.851792 26692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:08.871026 26692 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:08.871743 26692 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:08.875600 26692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Generated new TSK 0
I20260926 09:22:08.875875 26692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:08.909565 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41391
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:08.996588 26704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:08.996918 26704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:08.996965 26704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:08.999260 26704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:08.999343 26704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:08.999387 26704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:08.999425 26704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:09.002303 26704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:09.002646 26704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:09.002980 26704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:09.009714 26709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.009790 26712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.009714 26710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:09.009940 26704 server_base.cc:1034] running on GCE node
I20260926 09:22:09.010213 26704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:09.010496 26704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:09.011667 26704 hybrid_clock.cc:648] HybridClock initialized: now 1790414529011654 us; error 42 us; skew 500 ppm
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:09.014353 26704 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:09.014777 26704 webserver.cc:460] Webserver started at http://127.24.235.65:42713/ using document root <none> and password file <none>
I20260926 09:22:09.014971 26704 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:09.015012 26704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:09.015105 26704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:09.016273 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "9eba340f935141477f5dd2542d9de025"
server_key_iv: "fb882b22cd7f853e2715e9dedddc4b75"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.016585 26704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "9eba340f935141477f5dd2542d9de025"
server_key_iv: "fb882b22cd7f853e2715e9dedddc4b75"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.018050 26704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:09.018836 26719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.019042 26704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.019172 26704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "9eba340f935141477f5dd2542d9de025"
server_key_iv: "fb882b22cd7f853e2715e9dedddc4b75"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.019245 26704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:09.031265 26704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:09.033607 26704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:09.033767 26704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:09.034498 26704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:09.034557 26704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.034614 26704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:09.034651 26704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.041949 26704 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:42069
I20260926 09:22:09.042026 26834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:42069 every 8 connection(s)
I20260926 09:22:09.042377 26704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:09.045835 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26704
I20260926 09:22:09.045967 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:09.046007 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.035381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:34857 (local address 127.24.235.126:41391)
0926 09:22:09.035432 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.035435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.035542 (+ 107us) server_negotiation.cc:407] Connection header received
0926 09:22:09.036164 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.036165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.036181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.036233 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.037207 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.038021 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.039789 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.040078 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.040149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.043063 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.043065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.043065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.043074 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.044435 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.044872 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.044873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.044874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.044936 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.045223 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.045224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.045224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.045365 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.045441 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.045782 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.045901 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:22:09.046192 25517 external_mini_cluster.cc:1428] Setting key b4901e25b97b6b6d5577f87e07b7ca0f
I20260926 09:22:09.046200 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.035188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.65:34857)
0926 09:22:09.035375 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.035437 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.036032 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.036322 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.036326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.036744 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.037078 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.037086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.038160 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.038162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.039661 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.039666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.040185 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.040186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.040346 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.040992 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.041004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.042952 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.044552 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.044559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.044561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.044794 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.045043 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.045044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.045045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.045155 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.045432 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.045438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.045642 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.045748 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.045913 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260926 09:22:09.046834 26835 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41391
I20260926 09:22:09.046928 26835 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:09.048727 26835 heartbeater.cc:511] Master 127.24.235.126:41391 requested a full tablet report, sending...
I20260926 09:22:09.049240 26631 ts_manager.cc:194] Registered new tserver with Master: 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069)
I20260926 09:22:09.050144 26631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:34857
I20260926 09:22:09.054373 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.047830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:43117 (local address 127.24.235.126:41391)
0926 09:22:09.047873 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.047875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.047967 (+ 92us) server_negotiation.cc:407] Connection header received
0926 09:22:09.048035 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.048036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.048044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.048085 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.048557 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.049305 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.050370 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.050631 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.050675 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.051727 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.051728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.051728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.051734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.053193 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.053605 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.053606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.053606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.053670 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.053917 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.053918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.053919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.054030 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.054091 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.054251 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.054311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:09.054483 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.047750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.65:43117)
0926 09:22:09.047905 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.047917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.047980 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.048099 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.048099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.048379 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.048462 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.048464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.049480 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.049481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.050274 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.050276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.050750 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.050750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.050823 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.051293 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.051297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.051628 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.053267 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.053268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.053268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.053497 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.053746 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.053747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.053747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.053819 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.054102 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.054103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.054151 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.054236 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.054343 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:09.077592 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41391
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:09.162744 26844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:09.163017 26844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:09.163060 26844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:09.164289 26844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:09.164366 26844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:09.164402 26844 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:09.164454 26844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:09.166014 26844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:09.166324 26844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:09.166572 26844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:09.171967 26844 server_base.cc:1034] running on GCE node
W20260926 09:22:09.171918 26850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.171934 26852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.171936 26849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:09.172335 26844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:09.172631 26844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:09.173799 26844 hybrid_clock.cc:648] HybridClock initialized: now 1790414529173793 us; error 35 us; skew 500 ppm
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:09.176326 26844 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:09.176795 26844 webserver.cc:460] Webserver started at http://127.24.235.66:45629/ using document root <none> and password file <none>
I20260926 09:22:09.176998 26844 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:09.177099 26844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:09.177198 26844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:09.178221 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "61097ad98fb043a5991e40a3e8210075"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "b12bfb8eb94e07a6660d7b0943da1262"
server_key_iv: "4f3ebbebc71cf13c5b7b1a667846375f"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.178535 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "61097ad98fb043a5991e40a3e8210075"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "b12bfb8eb94e07a6660d7b0943da1262"
server_key_iv: "4f3ebbebc71cf13c5b7b1a667846375f"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.179853 26844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:09.180617 26859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.180806 26844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.180897 26844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "61097ad98fb043a5991e40a3e8210075"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "b12bfb8eb94e07a6660d7b0943da1262"
server_key_iv: "4f3ebbebc71cf13c5b7b1a667846375f"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.180953 26844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:09.196218 26844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:09.198690 26844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:09.198812 26844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:09.199307 26844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:09.199350 26844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.199376 26844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:09.199411 26844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.206652 26844 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:45871
I20260926 09:22:09.206741 26974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:45871 every 8 connection(s)
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:09.207060 26844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:09.210332 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.199824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:33009 (local address 127.24.235.126:41391)
0926 09:22:09.199871 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.199874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.200203 (+ 329us) server_negotiation.cc:407] Connection header received
0926 09:22:09.200967 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.200969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.200979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.201120 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.202063 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.203000 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.204536 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.204855 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.204947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.207511 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.207512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.207513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.207521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.208930 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.209386 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.209387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.209387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.209438 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.209770 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.209771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.209771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.209907 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.209983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.210196 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.210264 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:09.210485 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.199835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.66:33009)
0926 09:22:09.200042 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.200094 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.200835 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.201213 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.201232 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.201649 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.201955 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.201961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.203095 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.203096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.204449 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.204452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.204955 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.204957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.205140 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.205757 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.205766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.207395 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.209012 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.209063 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.209065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.209324 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.209479 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.209480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.209480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.209567 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.209981 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.209986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.210099 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.210178 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.210299 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260926 09:22:09.211136 26975 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41391
I20260926 09:22:09.211220 26975 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:09.211411 26975 heartbeater.cc:511] Master 127.24.235.126:41391 requested a full tablet report, sending...
I20260926 09:22:09.211796 26630 ts_manager.cc:194] Registered new tserver with Master: 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871)
I20260926 09:22:09.212339 26630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:33009
I20260926 09:22:09.213070 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26844
I20260926 09:22:09.213169 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:09.213346 25517 external_mini_cluster.cc:1428] Setting key 9b01d1a493642d8c4c27512369f03848
I20260926 09:22:09.218808 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.212175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:45007 (local address 127.24.235.126:41391)
0926 09:22:09.212209 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.212211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.212340 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:22:09.212408 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.212409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.212422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.212474 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.212885 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.213719 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.214511 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.214822 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.214895 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.216008 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.216010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.216010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.216016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.217553 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.217980 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.217981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.217981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.218039 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.218251 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.218251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.218252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.218411 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.218465 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.218663 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.218730 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:22:09.218905 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.212114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.66:45007)
0926 09:22:09.212265 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.212275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.212338 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.212502 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.212503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.212722 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.212782 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.212784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.213810 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.213811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.214400 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.214402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.214903 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.214904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.214970 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.215456 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.215462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.215929 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.217639 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.217640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.217641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.217896 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.218104 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.218105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.218106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.218182 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.218478 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.218479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.218548 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.218639 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.218769 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:09.241529 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41391
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:09.326009 26984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:09.326217 26984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:09.326242 26984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:09.327461 26984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:09.327512 26984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:09.327533 26984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:09.327551 26984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:09.329754 26984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:09.330017 26984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:09.330271 26984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:09.335522 26990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.335534 26992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:09.335532 26989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:09.335717 26984 server_base.cc:1034] running on GCE node
I20260926 09:22:09.336041 26984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:09.336337 26984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:09.337508 26984 hybrid_clock.cc:648] HybridClock initialized: now 1790414529337496 us; error 35 us; skew 500 ppm
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:09.340054 26984 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:09.340483 26984 webserver.cc:460] Webserver started at http://127.24.235.67:36113/ using document root <none> and password file <none>
I20260926 09:22:09.340742 26984 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:09.340790 26984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:09.340916 26984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:09.342020 26984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "6d85535312bc40488dadcf61b319b12d"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "f62210c298fb943ab7249d9bdcbd90a4"
server_key_iv: "fa8d6be983af7bd6ab1535dbecb0c06c"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.342383 26984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "6d85535312bc40488dadcf61b319b12d"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "f62210c298fb943ab7249d9bdcbd90a4"
server_key_iv: "fa8d6be983af7bd6ab1535dbecb0c06c"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.343840 26984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:09.344744 26999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.344985 26984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.345155 26984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "6d85535312bc40488dadcf61b319b12d"
format_stamp: "Formatted at 2026-09-26 09:22:09 on dist-test-slave-7zlw"
server_key: "f62210c298fb943ab7249d9bdcbd90a4"
server_key_iv: "fa8d6be983af7bd6ab1535dbecb0c06c"
server_key_version: "encryptionkey@0"
I20260926 09:22:09.345230 26984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:09.355093 26984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:09.357479 26984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:09.357681 26984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:09.358208 26984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:09.358271 26984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.358316 26984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:09.358356 26984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:09.365394 26984 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:45133
I20260926 09:22:09.365482 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:45133 every 8 connection(s)
I20260926 09:22:09.365796 26984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:09.366623 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 26984
I20260926 09:22:09.366688 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:09.366858 25517 external_mini_cluster.cc:1428] Setting key dc083ae8b2d1be109d0eb7b1f697ba8e
I20260926 09:22:09.369730 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.358701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:48805 (local address 127.24.235.126:41391)
0926 09:22:09.358770 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.358773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.359007 (+ 234us) server_negotiation.cc:407] Connection header received
0926 09:22:09.359522 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.359523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.359538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.359612 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.360506 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.361375 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.363247 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.363569 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.363645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.366387 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.366389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.366389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.366396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.368101 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.368611 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.368612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.368613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.368707 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.369084 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.369085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.369085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.369235 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.369302 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.369589 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.369662 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:22:09.370042 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.358706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.67:48805)
0926 09:22:09.358867 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.358913 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.359407 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.359695 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.359699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.360064 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.360391 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.360397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.361477 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.361478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.363114 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.363118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.363675 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.363677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.363800 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.364509 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.364522 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.366290 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.368201 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.368207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.368209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.368484 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.368800 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.368801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.368801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.368923 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.369305 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.369311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.369445 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.369571 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.369721 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260926 09:22:09.370672 27115 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41391
I20260926 09:22:09.370764 27115 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:09.370952 27115 heartbeater.cc:511] Master 127.24.235.126:41391 requested a full tablet report, sending...
I20260926 09:22:09.371321 26631 ts_manager.cc:194] Registered new tserver with Master: 6d85535312bc40488dadcf61b319b12d (127.24.235.67:45133)
I20260926 09:22:09.371840 26631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:48805
I20260926 09:22:09.377719 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.371741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:40699 (local address 127.24.235.126:41391)
0926 09:22:09.371781 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.371783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.371882 (+ 99us) server_negotiation.cc:407] Connection header received
0926 09:22:09.371928 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.371929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.371939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.371989 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.372339 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.372805 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.373544 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.373839 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.373916 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.375008 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.375009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.375009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.375016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.376525 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.376887 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.376889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.376889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.376949 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.377186 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.377187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.377188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.377314 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.377384 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.377546 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.377618 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:09.377735 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.371675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.24.235.67:40699)
0926 09:22:09.371807 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.371819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.371878 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.372038 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.372039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.372209 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.372262 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.372263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.372897 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.372898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.373462 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.373464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.373915 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.373916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.373992 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.374576 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.374580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.374885 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.376617 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.376618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.376619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.376810 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.377055 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.377056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.377056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.377121 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.377386 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.377387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.377434 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.377507 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.377621 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 09:22:09.378677 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:09.402292 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.395003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.24.235.126:41391)
0926 09:22:09.395064 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.395066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.395074 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:09.395100 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.395101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.395110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.395168 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.395744 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.396237 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.396993 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.397227 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.397283 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.399638 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.399639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.399640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.399646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.400982 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.401443 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.401445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.401445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.401506 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.401754 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.401755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.401756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.401884 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.401988 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.402099 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.402183 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:22:09.403914 26630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:09.405548 26630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:09.412237 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.408081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.24.235.66:45871)
0926 09:22:09.408293 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.408299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.408308 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:09.408350 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.408352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.408369 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.408451 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.408708 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.409897 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411222 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.411775 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411856 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.411915 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.412087 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.412130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260926 09:22:09.412236 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.407870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:45871 (local address 127.0.0.1:50564)
0926 09:22:09.407981 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.408047 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.408126 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.408534 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.408537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.408540 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.408627 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.408629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.409977 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.409978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.411139 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.411141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.411868 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.411949 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.411963 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.412068 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.412159 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":16,"thread_start_us":67,"threads_started":1}
I20260926 09:22:09.412304 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.408316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.24.235.67:45133)
0926 09:22:09.408508 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.408512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.408521 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:09.408554 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.408556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.408571 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.408647 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.408897 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.409982 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411051 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.411790 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411859 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.411915 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.412152 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.412195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20260926 09:22:09.412515 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.408091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45133 (local address 127.0.0.1:54742)
0926 09:22:09.408318 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.408327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.408371 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.408706 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.408707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.408709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.408818 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.408820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.410056 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.410057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.410938 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.410940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.411950 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.412018 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.412023 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.412072 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.412430 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260926 09:22:09.412631 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.407879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42069 (local address 127.0.0.1:52954)
0926 09:22:09.408215 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.408227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.408280 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.408701 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.408702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.408704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.408781 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.408783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.409781 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.409782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.411296 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.411297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.412279 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.412280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.412460 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.412464 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.412503 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.412538 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":47,"thread_start_us":79,"threads_started":1}
I20260926 09:22:09.412922 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.408042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.24.235.65:42069)
0926 09:22:09.408403 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.408409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.408419 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:09.408460 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.408462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.408528 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.408616 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.408889 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.409695 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.411380 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.412127 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.412187 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.412750 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.412776 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.412822 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":73,"threads_started":1}
I20260926 09:22:09.413314 27049 tablet_service.cc:1467] Processing CreateTablet for tablet 2b120573eb054d9882c689849853d56a (DEFAULT_TABLE table=test-table [id=b67c68ab516b4182b9c3a0809ca2291f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.413363 26909 tablet_service.cc:1467] Processing CreateTablet for tablet 2b120573eb054d9882c689849853d56a (DEFAULT_TABLE table=test-table [id=b67c68ab516b4182b9c3a0809ca2291f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.413484 26754 tablet_service.cc:1467] Processing CreateTablet for tablet 2b120573eb054d9882c689849853d56a (DEFAULT_TABLE table=test-table [id=b67c68ab516b4182b9c3a0809ca2291f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.413632 27049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b120573eb054d9882c689849853d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.413712 26909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b120573eb054d9882c689849853d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.413765 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b120573eb054d9882c689849853d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.417212 27138 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Bootstrap starting.
I20260926 09:22:09.417524 27139 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Bootstrap starting.
I20260926 09:22:09.418141 27139 tablet_bootstrap.cc:654] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.418154 27138 tablet_bootstrap.cc:654] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.418380 27139 log.cc:824] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:09.418488 27138 log.cc:824] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:09.418689 27140 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Bootstrap starting.
I20260926 09:22:09.419130 27139 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: No bootstrap required, opened a new log
I20260926 09:22:09.419203 27139 ts_tablet_manager.cc:1397] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:09.419230 27138 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: No bootstrap required, opened a new log
I20260926 09:22:09.419293 27138 ts_tablet_manager.cc:1397] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:09.419312 27140 tablet_bootstrap.cc:654] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.419620 27140 log.cc:824] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:09.420408 27140 tablet_bootstrap.cc:492] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: No bootstrap required, opened a new log
I20260926 09:22:09.420488 27140 ts_tablet_manager.cc:1397] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:09.421219 27138 raft_consensus.cc:348] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.421370 27138 raft_consensus.cc:374] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.421411 27138 raft_consensus.cc:729] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d85535312bc40488dadcf61b319b12d, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.421345 27139 raft_consensus.cc:348] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.421491 27139 raft_consensus.cc:374] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.421535 27139 raft_consensus.cc:729] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f20ddcfcd543feb0a9ed2dc2e722bd, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.421591 27138 consensus_queue.cc:260] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.421661 27139 consensus_queue.cc:260] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.421895 27138 ts_tablet_manager.cc:1428] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:09.421941 27139 ts_tablet_manager.cc:1428] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:09.422019 27115 heartbeater.cc:503] Master 127.24.235.126:41391 was elected leader, sending a full tablet report...
I20260926 09:22:09.422060 26835 heartbeater.cc:503] Master 127.24.235.126:41391 was elected leader, sending a full tablet report...
I20260926 09:22:09.422540 27140 raft_consensus.cc:348] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.422681 27140 raft_consensus.cc:374] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.422725 27140 raft_consensus.cc:729] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61097ad98fb043a5991e40a3e8210075, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.422865 27140 consensus_queue.cc:260] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.423082 27140 ts_tablet_manager.cc:1428] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:09.423164 26975 heartbeater.cc:503] Master 127.24.235.126:41391 was elected leader, sending a full tablet report...
W20260926 09:22:09.457672 26976 tablet.cc:2369] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:22:09.543350 26836 tablet.cc:2369] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:22:09.616524 27116 tablet.cc:2369] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:09.680742 27146 raft_consensus.cc:482] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:09.680876 27146 raft_consensus.cc:504] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.681423 27146 leader_election.cc:288] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 6d85535312bc40488dadcf61b319b12d (127.24.235.67:45133)
I20260926 09:22:09.684340 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.681762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:51715 (local address 127.24.235.65:42069)
0926 09:22:09.681823 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.681825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.681837 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:09.681872 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.681873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.681883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.681961 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.682129 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.682616 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.683628 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.683983 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.684063 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.684073 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.684234 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.684271 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:22:09.684340 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.681686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42069 (local address 127.24.235.66:51715)
0926 09:22:09.681747 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.681760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.681808 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.681970 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.681972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.681974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.682045 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.682046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.682727 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.682728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.683532 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.683534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.684059 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.684060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.684141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.684144 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.684225 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.684263 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:22:09.684760 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b120573eb054d9882c689849853d56a" candidate_uuid: "61097ad98fb043a5991e40a3e8210075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" is_pre_election: true
I20260926 09:22:09.684870 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.681814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45133 (local address 127.24.235.66:35975)
0926 09:22:09.681910 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.681919 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.681957 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.682106 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.682107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.682109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.682304 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.682306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.682942 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.682943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.683853 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.683856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.684567 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.684568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.684651 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.684654 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.684741 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.684780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260926 09:22:09.684856 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.681910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:35975 (local address 127.24.235.67:45133)
0926 09:22:09.681977 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.681980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.681989 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:09.682017 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.682019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.682027 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.682096 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.682383 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.682859 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.683949 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.684386 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.684470 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.684479 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.684755 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.684789 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:22:09.684947 26789 raft_consensus.cc:2454] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61097ad98fb043a5991e40a3e8210075 in term 0.
I20260926 09:22:09.685329 26863 leader_election.cc:302] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f20ddcfcd543feb0a9ed2dc2e722bd, 61097ad98fb043a5991e40a3e8210075; no voters:
I20260926 09:22:09.685494 27146 raft_consensus.cc:2781] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:09.685606 27146 raft_consensus.cc:482] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:09.685645 27146 raft_consensus.cc:3037] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.685657 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b120573eb054d9882c689849853d56a" candidate_uuid: "61097ad98fb043a5991e40a3e8210075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d85535312bc40488dadcf61b319b12d" is_pre_election: true
I20260926 09:22:09.685824 27069 raft_consensus.cc:2454] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61097ad98fb043a5991e40a3e8210075 in term 0.
I20260926 09:22:09.686656 27146 raft_consensus.cc:504] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.686817 27146 leader_election.cc:288] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [CANDIDATE]: Term 1 election: Requested vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 6d85535312bc40488dadcf61b319b12d (127.24.235.67:45133)
I20260926 09:22:09.687003 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b120573eb054d9882c689849853d56a" candidate_uuid: "61097ad98fb043a5991e40a3e8210075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d85535312bc40488dadcf61b319b12d"
I20260926 09:22:09.686983 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b120573eb054d9882c689849853d56a" candidate_uuid: "61097ad98fb043a5991e40a3e8210075" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
I20260926 09:22:09.687073 27069 raft_consensus.cc:3037] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.687096 26789 raft_consensus.cc:3037] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.687825 27069 raft_consensus.cc:2454] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61097ad98fb043a5991e40a3e8210075 in term 1.
I20260926 09:22:09.687827 26789 raft_consensus.cc:2454] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61097ad98fb043a5991e40a3e8210075 in term 1.
I20260926 09:22:09.687999 26863 leader_election.cc:302] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f20ddcfcd543feb0a9ed2dc2e722bd, 61097ad98fb043a5991e40a3e8210075; no voters:
I20260926 09:22:09.688143 27146 raft_consensus.cc:2781] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:09.688349 27146 raft_consensus.cc:686] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 1 LEADER]: Becoming Leader. State: Replica: 61097ad98fb043a5991e40a3e8210075, State: Running, Role: LEADER
I20260926 09:22:09.688464 27146 consensus_queue.cc:237] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.689404 26631 catalog_manager.cc:5495] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61097ad98fb043a5991e40a3e8210075 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "61097ad98fb043a5991e40a3e8210075" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:09.765367 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.762734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.24.235.66:45871)
0926 09:22:09.762820 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.762822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.762835 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:09.762879 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.762881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.762907 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.762974 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:09.763131 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.763609 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.764439 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.764634 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.764712 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.764956 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.765098 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.765201 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.765244 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.765269 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 09:22:09.767272 27069 raft_consensus.cc:1261] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Refusing update from remote peer 61097ad98fb043a5991e40a3e8210075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.767578 27146 consensus_queue.cc:1035] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.769119 26789 raft_consensus.cc:1261] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Refusing update from remote peer 61097ad98fb043a5991e40a3e8210075: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.769392 27148 consensus_queue.cc:1035] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.770144 27151 mvcc.cc:205] Tried to move back new op lower bound from 7333537913923874816 to 7333537913604780032. Current Snapshot: MvccSnapshot[applied={T|T < 7333537913923874816}]
I20260926 09:22:09.772331 27156 mvcc.cc:205] Tried to move back new op lower bound from 7333537913923874816 to 7333537913604780032. Current Snapshot: MvccSnapshot[applied={T|T < 7333537913923874816}]
I20260926 09:22:09.775010 26631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:09.775126 26631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:09.775936 26631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Sending DeleteTablet for 3 replicas of tablet 2b120573eb054d9882c689849853d56a
I20260926 09:22:09.776307 26909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b120573eb054d9882c689849853d56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50564
I20260926 09:22:09.776305 26754 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b120573eb054d9882c689849853d56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:52954
I20260926 09:22:09.776512 27049 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b120573eb054d9882c689849853d56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:54742
I20260926 09:22:09.776749 27161 tablet_replica.cc:330] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: stopping tablet replica
I20260926 09:22:09.776785 27160 tablet_replica.cc:330] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: stopping tablet replica
I20260926 09:22:09.776853 27161 raft_consensus.cc:2229] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:09.776796 26631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 09:22:09.776885 27160 raft_consensus.cc:2229] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 09:22:09.776921 26631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:09.776950 27161 raft_consensus.cc:2258] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.776976 27160 raft_consensus.cc:2258] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.777544 27162 tablet_replica.cc:330] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: stopping tablet replica
I20260926 09:22:09.777572 27161 ts_tablet_manager.cc:1905] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.777725 27162 raft_consensus.cc:2229] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:09.777902 27162 raft_consensus.cc:2258] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.778537 26909 tablet_service.cc:1467] Processing CreateTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e (DEFAULT_TABLE table=test-table [id=a8391900c80d4fea97aeda9fdfc55b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.778671 26909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e137a82bbb046e2a4d027a866cdf17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.778687 26754 tablet_service.cc:1467] Processing CreateTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e (DEFAULT_TABLE table=test-table [id=a8391900c80d4fea97aeda9fdfc55b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.778780 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e137a82bbb046e2a4d027a866cdf17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.779649 27161 ts_tablet_manager.cc:1918] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:09.779716 27161 log.cc:1196] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/2b120573eb054d9882c689849853d56a
I20260926 09:22:09.779974 27161 ts_tablet_manager.cc:1939] T 2b120573eb054d9882c689849853d56a P 61097ad98fb043a5991e40a3e8210075: Deleting consensus metadata
I20260926 09:22:09.780453 27140 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Bootstrap starting.
I20260926 09:22:09.780797 27139 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Bootstrap starting.
I20260926 09:22:09.780874 27140 tablet_bootstrap.cc:654] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.781378 27049 tablet_service.cc:1467] Processing CreateTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e (DEFAULT_TABLE table=test-table [id=a8391900c80d4fea97aeda9fdfc55b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:09.781509 27049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e137a82bbb046e2a4d027a866cdf17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.781721 27139 tablet_bootstrap.cc:654] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.781805 27140 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: No bootstrap required, opened a new log
I20260926 09:22:09.781888 27140 ts_tablet_manager.cc:1397] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.782070 27140 raft_consensus.cc:348] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.782186 27140 raft_consensus.cc:374] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.782219 27140 raft_consensus.cc:729] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61097ad98fb043a5991e40a3e8210075, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.782275 27140 consensus_queue.cc:260] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.782383 27140 ts_tablet_manager.cc:1428] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:09.783125 27139 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: No bootstrap required, opened a new log
I20260926 09:22:09.783191 27139 ts_tablet_manager.cc:1397] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:09.783380 27139 raft_consensus.cc:348] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.783475 27139 raft_consensus.cc:374] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.783499 27139 raft_consensus.cc:729] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f20ddcfcd543feb0a9ed2dc2e722bd, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.783572 27139 consensus_queue.cc:260] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.783634 27138 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Bootstrap starting.
I20260926 09:22:09.783699 27139 ts_tablet_manager.cc:1428] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:09.784430 27138 tablet_bootstrap.cc:654] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.785271 26618 catalog_manager.cc:4841] TS 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871): tablet 2b120573eb054d9882c689849853d56a (table test-table [id=b67c68ab516b4182b9c3a0809ca2291f]) successfully deleted
I20260926 09:22:09.785575 27138 tablet_bootstrap.cc:492] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: No bootstrap required, opened a new log
I20260926 09:22:09.785624 27138 ts_tablet_manager.cc:1397] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:22:09.785992 27162 ts_tablet_manager.cc:1905] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.786047 27160 ts_tablet_manager.cc:1905] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.786100 27138 raft_consensus.cc:348] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.786185 27138 raft_consensus.cc:374] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.786207 27138 raft_consensus.cc:729] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d85535312bc40488dadcf61b319b12d, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.786258 27138 consensus_queue.cc:260] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.786341 27138 ts_tablet_manager.cc:1428] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:09.787997 27162 ts_tablet_manager.cc:1918] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:09.788110 27160 ts_tablet_manager.cc:1918] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:09.788141 27162 log.cc:1196] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/2b120573eb054d9882c689849853d56a
I20260926 09:22:09.788208 27160 log.cc:1196] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/2b120573eb054d9882c689849853d56a
I20260926 09:22:09.788445 27162 ts_tablet_manager.cc:1939] T 2b120573eb054d9882c689849853d56a P 6d85535312bc40488dadcf61b319b12d: Deleting consensus metadata
I20260926 09:22:09.788503 27160 ts_tablet_manager.cc:1939] T 2b120573eb054d9882c689849853d56a P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting consensus metadata
I20260926 09:22:09.788894 26619 catalog_manager.cc:4841] TS 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069): tablet 2b120573eb054d9882c689849853d56a (table test-table [id=b67c68ab516b4182b9c3a0809ca2291f]) successfully deleted
I20260926 09:22:09.788983 26619 catalog_manager.cc:4841] TS 6d85535312bc40488dadcf61b319b12d (127.24.235.67:45133): tablet 2b120573eb054d9882c689849853d56a (table test-table [id=b67c68ab516b4182b9c3a0809ca2291f]) successfully deleted
I20260926 09:22:09.824767 27144 raft_consensus.cc:482] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:09.824891 27144 raft_consensus.cc:504] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.825389 27144 leader_election.cc:288] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871)
I20260926 09:22:09.829396 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.825585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:45871 (local address 127.24.235.67:37779)
0926 09:22:09.825660 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.825672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.825718 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.825893 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.825894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.825897 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.825971 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.825973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.826818 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.826819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.828107 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.828109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.828886 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.828887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.829068 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.829074 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.829258 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.829312 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260926 09:22:09.829396 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.825712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37779 (local address 127.24.235.66:45871)
0926 09:22:09.825757 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.825760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.825768 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:09.825796 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.825797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.825810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.825867 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.826062 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.826722 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.828231 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.828799 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.828882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.828891 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.829286 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.829327 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:09.829456 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.825720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:57763 (local address 127.24.235.65:42069)
0926 09:22:09.825765 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.825767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.825967 (+ 200us) server_negotiation.cc:407] Connection header received
0926 09:22:09.826027 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.826028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.826038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.826081 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:09.826254 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.826990 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.828558 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.828999 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.829097 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.829106 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.829333 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.829382 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:09.829737 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.825594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42069 (local address 127.24.235.67:57763)
0926 09:22:09.825891 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.825901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.825973 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.826098 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.826099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.826100 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:09.826159 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.826160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.827079 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.827086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.828463 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.828465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.829138 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.829140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.829219 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.829223 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.829581 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.829630 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":188,"thread_start_us":58,"threads_started":1}
I20260926 09:22:09.830013 26929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e137a82bbb046e2a4d027a866cdf17e" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61097ad98fb043a5991e40a3e8210075" is_pre_election: true
I20260926 09:22:09.830083 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e137a82bbb046e2a4d027a866cdf17e" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" is_pre_election: true
I20260926 09:22:09.830164 26929 raft_consensus.cc:2454] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 0.
I20260926 09:22:09.830186 26789 raft_consensus.cc:2454] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 0.
I20260926 09:22:09.830420 27002 leader_election.cc:302] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61097ad98fb043a5991e40a3e8210075, 6d85535312bc40488dadcf61b319b12d; no voters:
I20260926 09:22:09.830603 27144 raft_consensus.cc:2781] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:09.830668 27144 raft_consensus.cc:482] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:09.830703 27144 raft_consensus.cc:3037] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.831547 27144 raft_consensus.cc:504] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.831722 27144 leader_election.cc:288] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 election: Requested vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871)
I20260926 09:22:09.831918 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e137a82bbb046e2a4d027a866cdf17e" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
I20260926 09:22:09.831913 26929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e137a82bbb046e2a4d027a866cdf17e" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61097ad98fb043a5991e40a3e8210075"
I20260926 09:22:09.832002 26929 raft_consensus.cc:3037] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.832006 26789 raft_consensus.cc:3037] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.832664 26789 raft_consensus.cc:2454] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 1.
I20260926 09:22:09.832782 26929 raft_consensus.cc:2454] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 1.
I20260926 09:22:09.832857 27003 leader_election.cc:302] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f20ddcfcd543feb0a9ed2dc2e722bd, 6d85535312bc40488dadcf61b319b12d; no voters:
I20260926 09:22:09.832978 27144 raft_consensus.cc:2781] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:09.833216 27144 raft_consensus.cc:686] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 1 LEADER]: Becoming Leader. State: Replica: 6d85535312bc40488dadcf61b319b12d, State: Running, Role: LEADER
I20260926 09:22:09.833319 27144 consensus_queue.cc:237] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.834288 26631 catalog_manager.cc:5495] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d85535312bc40488dadcf61b319b12d (127.24.235.67). New cstate: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } health_report { overall_health: HEALTHY } } }
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414528, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:09.865945 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.857943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.24.235.126:41391)
0926 09:22:09.857998 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.858000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.858193 (+ 193us) server_negotiation.cc:407] Connection header received
0926 09:22:09.858277 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.858278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.858289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.858359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.858868 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.859378 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.860082 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.860268 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.860348 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.862872 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.862873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.862874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.862882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.864508 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.864991 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.864992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.864993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.865122 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.865313 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.865314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.865315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.865436 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.865547 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.865714 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.865810 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":29}
I20260926 09:22:09.866001 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.857884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41391 (local address 127.0.0.1:46566)
0926 09:22:09.858097 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.858107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.858213 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.858363 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.858364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.858722 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:09.858774 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.858778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.859457 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.859458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.859988 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.859990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.860350 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.860351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.860401 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.861008 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:09.861052 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:09.862748 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:09.864674 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.864683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.864686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.864901 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.865188 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:09.865189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:09.865190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:09.865252 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:09.865537 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:09.865546 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:09.865603 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.865705 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.865863 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":44,"thread_start_us":140,"threads_started":1}
I20260926 09:22:09.868882 26631 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:46566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:22:09.869177 26631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:09.870710 26909 tablet_service.cc:1467] Processing CreateTablet for tablet c138e65d5f544297ad79048316d6d56b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091b0954943a4cbc922df31fbb3817ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:09.870814 26754 tablet_service.cc:1467] Processing CreateTablet for tablet c138e65d5f544297ad79048316d6d56b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091b0954943a4cbc922df31fbb3817ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:09.870893 26909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c138e65d5f544297ad79048316d6d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.870992 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c138e65d5f544297ad79048316d6d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.872519 27049 tablet_service.cc:1467] Processing CreateTablet for tablet c138e65d5f544297ad79048316d6d56b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091b0954943a4cbc922df31fbb3817ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:09.872679 27049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c138e65d5f544297ad79048316d6d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:09.874666 27140 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: Bootstrap starting.
I20260926 09:22:09.874811 27138 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: Bootstrap starting.
I20260926 09:22:09.875084 27140 tablet_bootstrap.cc:654] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.875219 27138 tablet_bootstrap.cc:654] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.875789 27138 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: No bootstrap required, opened a new log
I20260926 09:22:09.875851 27138 ts_tablet_manager.cc:1397] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:09.875907 27139 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: Bootstrap starting.
I20260926 09:22:09.876011 27138 raft_consensus.cc:348] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.876096 27138 raft_consensus.cc:374] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.876127 27138 raft_consensus.cc:729] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d85535312bc40488dadcf61b319b12d, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.876169 27138 consensus_queue.cc:260] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.876264 27138 ts_tablet_manager.cc:1428] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:09.876348 27139 tablet_bootstrap.cc:654] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:09.876384 27144 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.876477 27144 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.877337 27139 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: No bootstrap required, opened a new log
I20260926 09:22:09.877398 27139 ts_tablet_manager.cc:1397] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:09.877576 27139 raft_consensus.cc:348] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.877702 27139 raft_consensus.cc:374] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.877743 27139 raft_consensus.cc:729] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f20ddcfcd543feb0a9ed2dc2e722bd, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.877791 27139 consensus_queue.cc:260] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.877959 27139 ts_tablet_manager.cc:1428] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.878084 27145 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.878190 27145 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.878446 27140 tablet_bootstrap.cc:492] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: No bootstrap required, opened a new log
I20260926 09:22:09.878510 27140 ts_tablet_manager.cc:1397] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 09:22:09.879071 27140 raft_consensus.cc:348] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.879211 27140 raft_consensus.cc:374] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:09.879236 27140 raft_consensus.cc:729] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61097ad98fb043a5991e40a3e8210075, State: Initialized, Role: FOLLOWER
I20260926 09:22:09.879288 27140 consensus_queue.cc:260] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.879405 27140 ts_tablet_manager.cc:1428] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:09.879443 27148 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.879634 27148 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.904484 27144 raft_consensus.cc:482] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:09.904598 27144 raft_consensus.cc:504] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.904793 27144 leader_election.cc:288] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871)
I20260926 09:22:09.904989 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c138e65d5f544297ad79048316d6d56b" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" is_pre_election: true
I20260926 09:22:09.905150 26789 raft_consensus.cc:2454] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 0.
I20260926 09:22:09.905092 26929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c138e65d5f544297ad79048316d6d56b" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61097ad98fb043a5991e40a3e8210075" is_pre_election: true
I20260926 09:22:09.905233 26929 raft_consensus.cc:2454] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 0.
I20260926 09:22:09.905381 27003 leader_election.cc:302] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f20ddcfcd543feb0a9ed2dc2e722bd, 6d85535312bc40488dadcf61b319b12d; no voters:
I20260926 09:22:09.905473 27144 raft_consensus.cc:2781] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:09.905515 27144 raft_consensus.cc:482] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:09.905534 27144 raft_consensus.cc:3037] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.906236 27144 raft_consensus.cc:504] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.906383 27144 leader_election.cc:288] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 election: Requested vote from peers 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069), 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871)
I20260926 09:22:09.906531 26789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c138e65d5f544297ad79048316d6d56b" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd"
I20260926 09:22:09.906556 26929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c138e65d5f544297ad79048316d6d56b" candidate_uuid: "6d85535312bc40488dadcf61b319b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61097ad98fb043a5991e40a3e8210075"
I20260926 09:22:09.906620 26929 raft_consensus.cc:3037] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.906620 26789 raft_consensus.cc:3037] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:09.907385 26929 raft_consensus.cc:2454] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 1.
I20260926 09:22:09.907385 26789 raft_consensus.cc:2454] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d85535312bc40488dadcf61b319b12d in term 1.
I20260926 09:22:09.907558 27002 leader_election.cc:302] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61097ad98fb043a5991e40a3e8210075, 6d85535312bc40488dadcf61b319b12d; no voters:
I20260926 09:22:09.907685 27144 raft_consensus.cc:2781] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:09.907760 27144 raft_consensus.cc:686] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [term 1 LEADER]: Becoming Leader. State: Replica: 6d85535312bc40488dadcf61b319b12d, State: Running, Role: LEADER
I20260926 09:22:09.907819 27144 consensus_queue.cc:237] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } }
I20260926 09:22:09.908264 27167 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: TxnStatusTablet state changed. Reason: New leader 6d85535312bc40488dadcf61b319b12d. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.908362 27167 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:09.908565 27179 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:22:09.908735 27179 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:22:09.908788 26630 catalog_manager.cc:5495] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d85535312bc40488dadcf61b319b12d (127.24.235.67). New cstate: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } health_report { overall_health: HEALTHY } } }
I20260926 09:22:09.923771 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.920491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.24.235.67:45133)
0926 09:22:09.920531 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.920533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.920540 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:22:09.920583 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.920583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.920611 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.920669 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:09.920844 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.921731 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.922847 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.923129 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.923209 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.923409 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.923544 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.923610 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.923685 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.923708 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:09.923815 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.920233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45133 (local address 127.0.0.1:54758)
0926 09:22:09.920353 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.920365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.920406 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.920721 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.920722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.920723 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:09.920775 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.920777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.921832 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.921833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.922765 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.922768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.923227 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.923228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.923292 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.923311 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.923616 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.923624 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.923677 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.923729 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 09:22:09.925237 26789 raft_consensus.cc:1261] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Refusing update from remote peer 6d85535312bc40488dadcf61b319b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.925351 26929 raft_consensus.cc:1261] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Refusing update from remote peer 6d85535312bc40488dadcf61b319b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.925511 27144 consensus_queue.cc:1035] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.925614 27167 consensus_queue.cc:1035] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.926374 27145 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: TxnStatusTablet state changed. Reason: New leader 6d85535312bc40488dadcf61b319b12d. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.926402 27148 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: TxnStatusTablet state changed. Reason: New leader 6d85535312bc40488dadcf61b319b12d. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.926476 27145 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.926491 27148 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.927403 27167 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.927485 27167 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:09.927793 27145 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.927871 27145 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 39f20ddcfcd543feb0a9ed2dc2e722bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.929291 27144 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.929389 27144 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 6d85535312bc40488dadcf61b319b12d: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:09.929730 27148 tablet_replica.cc:439] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d85535312bc40488dadcf61b319b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 } } peers { permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 } } peers { permanent_uuid: "6d85535312bc40488dadcf61b319b12d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45133 } } }
I20260926 09:22:09.929818 27148 tablet_replica.cc:442] T c138e65d5f544297ad79048316d6d56b P 61097ad98fb043a5991e40a3e8210075: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:09.932282 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.929295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.24.235.67:45133)
0926 09:22:09.929485 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.929487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.929495 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:09.929516 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.929516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.929525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.929590 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:09.929905 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.930665 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.931447 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.931753 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.931822 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.931938 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.931973 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.932027 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.932069 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.932089 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260926 09:22:09.937704 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.934918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:51149 (local address 127.24.235.67:45133)
0926 09:22:09.934952 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.934954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.935160 (+ 206us) server_negotiation.cc:407] Connection header received
0926 09:22:09.935205 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.935206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.935216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.935277 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:09.935480 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.936190 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.936928 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.937259 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.937330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.937474 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.937506 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.937558 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.937604 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.937622 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:22:09.937813 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.934863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45133 (local address 127.24.235.67:51149)
0926 09:22:09.935094 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:09.935104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:09.935155 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:09.935285 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:09.935286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:09.935287 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:09.935422 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.935423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.936271 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.936272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.936853 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:09.936855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.937330 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.937331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:09.937381 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.937396 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:09.937564 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:09.937565 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:22:09.937601 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 09:22:09.937635 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260926 09:22:09.940238 26789 raft_consensus.cc:1261] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Refusing update from remote peer 6d85535312bc40488dadcf61b319b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.940348 26929 raft_consensus.cc:1261] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Refusing update from remote peer 6d85535312bc40488dadcf61b319b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:09.940471 27188 consensus_queue.cc:1035] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f20ddcfcd543feb0a9ed2dc2e722bd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.940586 27187 consensus_queue.cc:1035] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61097ad98fb043a5991e40a3e8210075" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:09.941722 27152 mvcc.cc:205] Tried to move back new op lower bound from 7333537914633777152 to 7333537914198597632. Current Snapshot: MvccSnapshot[applied={T|T < 7333537914633777152}]
I20260926 09:22:09.953338 26631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:09.953450 26631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:09.954135 26631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f193c876954f470797ecde266f7a7ee5: Sending DeleteTablet for 3 replicas of tablet 3e137a82bbb046e2a4d027a866cdf17e
I20260926 09:22:09.954399 26909 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50564
I20260926 09:22:09.954442 26754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:52954
I20260926 09:22:09.954558 27161 tablet_replica.cc:330] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: stopping tablet replica
I20260926 09:22:09.954559 27160 tablet_replica.cc:330] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: stopping tablet replica
I20260926 09:22:09.954627 27160 raft_consensus.cc:2229] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:09.954627 27161 raft_consensus.cc:2229] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:09.954674 27160 raft_consensus.cc:2258] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.954674 27161 raft_consensus.cc:2258] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.955015 27160 ts_tablet_manager.cc:1905] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.955147 27161 ts_tablet_manager.cc:1905] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.956591 27049 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e137a82bbb046e2a4d027a866cdf17e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:09 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:54742
I20260926 09:22:09.956787 27162 tablet_replica.cc:330] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: stopping tablet replica
I20260926 09:22:09.956852 27160 ts_tablet_manager.cc:1918] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:09.956862 27162 raft_consensus.cc:2229] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:09.956905 27160 log.cc:1196] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/3e137a82bbb046e2a4d027a866cdf17e
I20260926 09:22:09.956983 27162 raft_consensus.cc:2258] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:09.957196 27160 ts_tablet_manager.cc:1939] T 3e137a82bbb046e2a4d027a866cdf17e P 39f20ddcfcd543feb0a9ed2dc2e722bd: Deleting consensus metadata
I20260926 09:22:09.957181 27161 ts_tablet_manager.cc:1918] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:09.957269 27161 log.cc:1196] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/3e137a82bbb046e2a4d027a866cdf17e
I20260926 09:22:09.957361 27162 ts_tablet_manager.cc:1905] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:09.957492 27161 ts_tablet_manager.cc:1939] T 3e137a82bbb046e2a4d027a866cdf17e P 61097ad98fb043a5991e40a3e8210075: Deleting consensus metadata
I20260926 09:22:09.957844 26618 catalog_manager.cc:4841] TS 61097ad98fb043a5991e40a3e8210075 (127.24.235.66:45871): tablet 3e137a82bbb046e2a4d027a866cdf17e (table test-table [id=a8391900c80d4fea97aeda9fdfc55b48]) successfully deleted
I20260926 09:22:09.959162 27162 ts_tablet_manager.cc:1918] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:09.959216 27162 log.cc:1196] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/3e137a82bbb046e2a4d027a866cdf17e
I20260926 09:22:09.959410 27162 ts_tablet_manager.cc:1939] T 3e137a82bbb046e2a4d027a866cdf17e P 6d85535312bc40488dadcf61b319b12d: Deleting consensus metadata
I20260926 09:22:09.959602 26619 catalog_manager.cc:4841] TS 39f20ddcfcd543feb0a9ed2dc2e722bd (127.24.235.65:42069): tablet 3e137a82bbb046e2a4d027a866cdf17e (table test-table [id=a8391900c80d4fea97aeda9fdfc55b48]) successfully deleted
I20260926 09:22:09.959801 26619 catalog_manager.cc:4841] TS 6d85535312bc40488dadcf61b319b12d (127.24.235.67:45133): tablet 3e137a82bbb046e2a4d027a866cdf17e (table test-table [id=a8391900c80d4fea97aeda9fdfc55b48]) successfully deleted
Sep 26 09:22:09 dist-test-slave-7zlw krb5kdc[26587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:09.968592 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.956824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.24.235.65:42069)
0926 09:22:09.956869 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.956872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.958012 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 09:22:09.958079 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.958080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.958095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.958157 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.959091 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.959821 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.960905 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.961397 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.961478 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.965162 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.965182 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.965182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.965195 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.967302 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.967722 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.967731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.967732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.967789 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.967991 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.967992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.967992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.968146 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.968313 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.968385 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.968474 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 09:22:09.968974 26829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52958
I20260926 09:22:09.976516 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:09.970129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.24.235.126:41391)
0926 09:22:09.970177 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:09.970179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:09.970242 (+ 63us) server_negotiation.cc:407] Connection header received
0926 09:22:09.970339 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:09.970339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:09.970351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:09.970398 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:09.970936 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.971437 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.972179 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:09.972389 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:09.972472 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:09.973788 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:09.973790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:09.973790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:09.973798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:09.975288 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.975645 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.975646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.975646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.975737 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:09.975942 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:09.975943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:09.975943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:09.976070 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:09.976170 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:09.976320 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 09:22:09.976407 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:22:09.976774 26631 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46568
I20260926 09:22:09.977670 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26704
I20260926 09:22:09.983453 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26844
I20260926 09:22:09.991086 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26984
I20260926 09:22:09.997330 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 26600
2026-09-26T09:22:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2459 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:10 dist-test-slave-7zlw krb5kdc[27199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:10 dist-test-slave-7zlw krb5kdc[27199](info): set up 2 sockets
Sep 26 09:22:10 dist-test-slave-7zlw krb5kdc[27199](info): commencing operation
krb5kdc: starting...
W20260926 09:22:11.072856 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:11.202205 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43675
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:11.286464 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:11.286741 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:11.286780 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:11.288079 27212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:11.288152 27212 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:22:11.288174 27212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:11.288228 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:11.288255 27212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:11.288278 27212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:11.289920 27212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:11.290218 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:11.290452 27212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:11.296031 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.296031 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:11.296197 27212 server_base.cc:1034] running on GCE node
W20260926 09:22:11.296034 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:11.296624 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:11.296960 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:11.298120 27212 hybrid_clock.cc:648] HybridClock initialized: now 1790414531298099 us; error 65 us; skew 500 ppm
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:11.300807 27212 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:11.301366 27212 webserver.cc:460] Webserver started at http://127.24.235.126:33187/ using document root <none> and password file <none>
I20260926 09:22:11.301668 27212 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:11.301744 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:11.301898 27212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:11.303017 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "90bd66cdd56c4f60ad47c22443425f2e"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "e6ffe7e307c083fc03b18ca0fbd4d34e"
server_key_iv: "52b20703522dcfaf287b422c27cb2947"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.303400 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "90bd66cdd56c4f60ad47c22443425f2e"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "e6ffe7e307c083fc03b18ca0fbd4d34e"
server_key_iv: "52b20703522dcfaf287b422c27cb2947"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.304849 27212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:11.305949 27227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.306167 27212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:11.306277 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "90bd66cdd56c4f60ad47c22443425f2e"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "e6ffe7e307c083fc03b18ca0fbd4d34e"
server_key_iv: "52b20703522dcfaf287b422c27cb2947"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.306365 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:11.330457 27212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:11.332770 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:11.332966 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:11.337937 27212 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:43675
I20260926 09:22:11.338119 27289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:43675 every 8 connection(s)
I20260926 09:22:11.338420 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:11.339224 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:11.341661 27290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Bootstrap starting.
I20260926 09:22:11.342309 27290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:11.342617 27290 log.cc:824] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:11.343397 27290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: No bootstrap required, opened a new log
I20260926 09:22:11.345834 27290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } }
I20260926 09:22:11.346001 27290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:11.346024 27290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bd66cdd56c4f60ad47c22443425f2e, State: Initialized, Role: FOLLOWER
I20260926 09:22:11.346202 27290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } }
I20260926 09:22:11.346293 27290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:11.346318 27290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:11.346383 27290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:11.347011 27290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } }
I20260926 09:22:11.347113 27290 leader_election.cc:302] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90bd66cdd56c4f60ad47c22443425f2e; no voters:
I20260926 09:22:11.347352 27290 leader_election.cc:288] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:11.347424 27293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:11.347651 27293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [term 1 LEADER]: Becoming Leader. State: Replica: 90bd66cdd56c4f60ad47c22443425f2e, State: Running, Role: LEADER
I20260926 09:22:11.347963 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27212
I20260926 09:22:11.348065 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:11.348258 25517 external_mini_cluster.cc:1428] Setting key ccd5cdc92deaa9d6299ba68ad1fef964
I20260926 09:22:11.348523 27293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } }
I20260926 09:22:11.347936 27290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:11.349522 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bd66cdd56c4f60ad47c22443425f2e. Latest consensus state: current_term: 1 leader_uuid: "90bd66cdd56c4f60ad47c22443425f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } } }
I20260926 09:22:11.349627 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:11.349980 27303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:11.349484 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90bd66cdd56c4f60ad47c22443425f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bd66cdd56c4f60ad47c22443425f2e" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43675 } } }
I20260926 09:22:11.350051 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:11.350633 27303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:11.352519 27303 catalog_manager.cc:1347] Generated new cluster ID: 8942f957f2ce4585a321bded13bc0623
I20260926 09:22:11.352583 27303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:11.360874 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.348920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.24.235.126:43675)
0926 09:22:11.349947 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.349959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.350545 (+ 586us) server_negotiation.cc:407] Connection header received
0926 09:22:11.350698 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.350706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.350769 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.350931 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.351744 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.352955 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.353845 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.354325 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.354455 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.357070 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.357076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.357076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.357102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.359000 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.359500 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.359503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.359504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.359634 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.359874 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.359875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.359876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.360185 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.360374 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.360507 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.360670 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":156,"threads_started":1}
I20260926 09:22:11.366665 27303 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:11.367224 27303 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:11.373065 27303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Generated new TSK 0
I20260926 09:22:11.373302 27303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:11.408043 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43675
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:11.500442 27316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:11.500689 27316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:11.500717 27316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:11.502092 27316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:11.502147 27316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:11.502171 27316 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:22:11.502184 27316 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:11.502233 27316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:11.503829 27316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:43675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:11.504133 27316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:11.504400 27316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:11.509711 27324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.509730 27322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.509884 27321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:11.510085 27316 server_base.cc:1034] running on GCE node
I20260926 09:22:11.510300 27316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:11.510635 27316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:11.511797 27316 hybrid_clock.cc:648] HybridClock initialized: now 1790414531511785 us; error 28 us; skew 500 ppm
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:11.514546 27316 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:11.515020 27316 webserver.cc:460] Webserver started at http://127.24.235.65:42071/ using document root <none> and password file <none>
I20260926 09:22:11.515215 27316 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:11.515257 27316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:11.515350 27316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:11.516444 27316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "c5922fa6d5e24027a5910e32ae038c4d"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9c043183377d2d68105493420e373cff"
server_key_iv: "1a4b87d8378bfaaf61c23d9015c01d8a"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.516759 27316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "c5922fa6d5e24027a5910e32ae038c4d"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9c043183377d2d68105493420e373cff"
server_key_iv: "1a4b87d8378bfaaf61c23d9015c01d8a"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.518265 27316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:11.519282 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.519551 27316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:11.519644 27316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "c5922fa6d5e24027a5910e32ae038c4d"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9c043183377d2d68105493420e373cff"
server_key_iv: "1a4b87d8378bfaaf61c23d9015c01d8a"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.519714 27316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:11.533882 27316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:11.536648 27316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:11.536820 27316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:11.537431 27316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:11.537488 27316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.537513 27316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:11.537526 27316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.544678 27316 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:38103
I20260926 09:22:11.544740 27446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:38103 every 8 connection(s)
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:11.545120 27316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:11.548350 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.537957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55453 (local address 127.24.235.126:43675)
0926 09:22:11.538034 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.538037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.538239 (+ 202us) server_negotiation.cc:407] Connection header received
0926 09:22:11.538759 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.538760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.538775 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.538857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.539791 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.540656 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.542273 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.542491 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.542551 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.545625 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.545627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.545627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.545638 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.546878 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.547307 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.547309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.547309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.547373 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.547682 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.547683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.547684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.547826 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.547907 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.548203 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.548284 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:11.548520 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.537917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.65:55453)
0926 09:22:11.538075 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.538141 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.538657 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.538946 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.538950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.539328 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.539668 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.539675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.540766 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.540767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.542159 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.542162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.542666 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.542667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.542836 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.543508 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.543520 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.545500 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.546957 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.546963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.546965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.547215 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.547465 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.547466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.547466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.547589 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.547910 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.547914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.548043 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.548134 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.548284 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 09:22:11.549190 27447 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43675
I20260926 09:22:11.549279 27447 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:11.549485 27447 heartbeater.cc:511] Master 127.24.235.126:43675 requested a full tablet report, sending...
I20260926 09:22:11.550026 27244 ts_manager.cc:194] Registered new tserver with Master: c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103)
I20260926 09:22:11.550912 27244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:55453
I20260926 09:22:11.554502 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27316
I20260926 09:22:11.554628 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:11.554839 25517 external_mini_cluster.cc:1428] Setting key b62e1ba91d5707423a7eb968241d16d5
I20260926 09:22:11.557099 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.550414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55433 (local address 127.24.235.126:43675)
0926 09:22:11.550469 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.550471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.550598 (+ 127us) server_negotiation.cc:407] Connection header received
0926 09:22:11.550646 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.550647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.550657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.550715 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.551015 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.551669 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.552652 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.552852 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.552915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.554047 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.554049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.554049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.554057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.555776 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.556181 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.556182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.556182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.556245 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.556433 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.556434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.556434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.556585 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.556666 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.556850 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.556943 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:11.557128 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.550352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.65:55433)
0926 09:22:11.550531 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.550541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.550591 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.550717 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.550717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.550885 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.550935 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.550937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.551921 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.551922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.552472 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.552475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.552945 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.552946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.553010 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.553591 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.553596 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.553906 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.555875 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.555876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.555877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.556096 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.556314 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.556314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.556315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.556372 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.556675 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.556676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.556725 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.556806 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.556916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:11.585306 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43675
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:11.667382 27456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:11.667613 27456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:11.667640 27456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:11.668915 27456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:11.668969 27456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:11.668992 27456 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:22:11.669004 27456 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:11.669090 27456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:11.670652 27456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:43675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:11.670926 27456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:11.671185 27456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:11.676383 27461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.676373 27464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.676373 27462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:11.676741 27456 server_base.cc:1034] running on GCE node
I20260926 09:22:11.676975 27456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:11.677268 27456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:11.678491 27456 hybrid_clock.cc:648] HybridClock initialized: now 1790414531678476 us; error 38 us; skew 500 ppm
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:11.681284 27456 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:11.681850 27456 webserver.cc:460] Webserver started at http://127.24.235.66:35977/ using document root <none> and password file <none>
I20260926 09:22:11.682127 27456 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:11.682197 27456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:11.682356 27456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:11.683490 27456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "5bd8e5e996443c4ab30835d4f8028692"
server_key_iv: "058bbc207b738e6b8e4866118995bc0f"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.683856 27456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "5bd8e5e996443c4ab30835d4f8028692"
server_key_iv: "058bbc207b738e6b8e4866118995bc0f"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.685316 27456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:11.686103 27471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.686269 27456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:11.686371 27456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "5bd8e5e996443c4ab30835d4f8028692"
server_key_iv: "058bbc207b738e6b8e4866118995bc0f"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.686450 27456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:11.702576 27456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:11.704733 27456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:11.704910 27456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:11.705583 27456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:11.705636 27456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.705667 27456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:11.705688 27456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:11.713822 27456 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:43455
I20260926 09:22:11.714289 27456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:11.714260 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:43455 every 8 connection(s)
I20260926 09:22:11.716542 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.705984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:42557 (local address 127.24.235.126:43675)
0926 09:22:11.706032 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.706035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.706269 (+ 234us) server_negotiation.cc:407] Connection header received
0926 09:22:11.706824 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.706825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.706839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.706890 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.707841 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.708450 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.709955 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.710360 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.710431 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.713202 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.713204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.713204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.713213 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.715039 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.715520 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.715522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.715522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.715586 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.715846 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.715847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.715848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.715972 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.716049 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.716365 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.716451 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:11.716739 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.705962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.66:42557)
0926 09:22:11.706119 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.706179 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.706706 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.707020 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.707025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.707368 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.707747 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.707755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.708554 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.708555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.709862 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.709865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.710438 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.710440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.710594 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.711226 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.711238 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.713115 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.715138 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.715144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.715147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.715429 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.715658 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.715659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.715660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.715771 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.716039 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.716046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.716217 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.716328 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.716499 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260926 09:22:11.717247 27587 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43675
I20260926 09:22:11.717334 27587 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:11.717535 27587 heartbeater.cc:511] Master 127.24.235.126:43675 requested a full tablet report, sending...
I20260926 09:22:11.717904 27244 ts_manager.cc:194] Registered new tserver with Master: e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455)
I20260926 09:22:11.718248 27244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:42557
I20260926 09:22:11.720872 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27456
I20260926 09:22:11.720990 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:11.721199 25517 external_mini_cluster.cc:1428] Setting key 71f2cfc3bc6e166099221ffed228acb8
I20260926 09:22:11.724403 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.718534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:45989 (local address 127.24.235.126:43675)
0926 09:22:11.718580 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.718582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.718590 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:11.718610 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.718611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.718620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.718668 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.718994 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.719484 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.720217 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.720421 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.720495 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.721670 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.721672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.721672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.721689 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.723172 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.723553 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.723555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.723555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.723617 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.723808 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.723809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.723810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.723975 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.724059 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.724224 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.724304 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:11.724409 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.718337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.66:45989)
0926 09:22:11.718483 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.718495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.718545 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.718671 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.718672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.718849 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.718907 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.718908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.719588 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.719589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.720119 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.720121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.720493 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.720494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.720570 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.721172 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.721177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.721568 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.723279 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.723280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.723280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.723462 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.723692 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.723693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.723693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.723757 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.724050 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.724051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.724102 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.724187 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.724306 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:11.748898 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43675
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:11.833206 27596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:11.833434 27596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:11.833462 27596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:11.834750 27596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:11.834800 27596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:11.834821 27596 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 09:22:11.834833 27596 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:11.834851 27596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:11.836285 27596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:43675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:11.836484 27596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:11.836683 27596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:11.841955 27601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.841931 27604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:11.841931 27602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:11.842301 27596 server_base.cc:1034] running on GCE node
I20260926 09:22:11.842604 27596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:11.842824 27596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:11.843994 27596 hybrid_clock.cc:648] HybridClock initialized: now 1790414531843976 us; error 46 us; skew 500 ppm
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:11.846653 27596 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:11.847107 27596 webserver.cc:460] Webserver started at http://127.24.235.67:32925/ using document root <none> and password file <none>
I20260926 09:22:11.847292 27596 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:11.847333 27596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:11.847422 27596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:11.848445 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "dcd8480b86f04a27a7563da34d1f9a69"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9b7d6fbefc366f4393131013d79010fb"
server_key_iv: "6e344f879e9735430ad0c3e805e6c631"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.848891 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "dcd8480b86f04a27a7563da34d1f9a69"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9b7d6fbefc366f4393131013d79010fb"
server_key_iv: "6e344f879e9735430ad0c3e805e6c631"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.850507 27596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:11.851312 27611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.851493 27596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:11.851574 27596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "dcd8480b86f04a27a7563da34d1f9a69"
format_stamp: "Formatted at 2026-09-26 09:22:11 on dist-test-slave-7zlw"
server_key: "9b7d6fbefc366f4393131013d79010fb"
server_key_iv: "6e344f879e9735430ad0c3e805e6c631"
server_key_version: "encryptionkey@0"
I20260926 09:22:11.851630 27596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:11.859259 27596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:11.861662 27596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:11.861773 27596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:11.862293 27596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:11.862334 27596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:11.862357 27596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:11.862370 27596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:11.870043 27596 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:44157
I20260926 09:22:11.870404 27596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:11.870755 27726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:44157 every 8 connection(s)
I20260926 09:22:11.873417 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.862738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37691 (local address 127.24.235.126:43675)
0926 09:22:11.862810 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.862812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.863079 (+ 267us) server_negotiation.cc:407] Connection header received
0926 09:22:11.863839 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.863841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.863857 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.863942 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.864922 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.865525 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.866647 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.866863 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.866954 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.870056 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.870058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.870058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.870068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.871786 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.872353 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.872354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.872355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.872418 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.872672 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.872673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.872673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.872873 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.872952 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.873266 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.873351 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:22:11.873572 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.862706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.67:37691)
0926 09:22:11.862918 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.862973 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.863693 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.864044 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.864049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.864478 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.864804 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.864811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.865641 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.865642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.866523 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.866525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.866964 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.866965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.867097 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.867737 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.867746 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.869919 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.871988 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.871994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.871996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.872253 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.872491 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.872492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.872492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.872599 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.872932 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.872937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.873128 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.873234 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.873367 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260926 09:22:11.874161 27727 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43675
I20260926 09:22:11.874267 27727 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:11.874490 27727 heartbeater.cc:511] Master 127.24.235.126:43675 requested a full tablet report, sending...
I20260926 09:22:11.874835 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27596
I20260926 09:22:11.874933 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:11.875123 25517 external_mini_cluster.cc:1428] Setting key b1574594d61c4569b9393a39fdba3ad1
I20260926 09:22:11.875574 27244 ts_manager.cc:194] Registered new tserver with Master: dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:11.876019 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:11.876039 27244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:37691
I20260926 09:22:11.881590 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.875371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:52989 (local address 127.24.235.126:43675)
0926 09:22:11.875415 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.875418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.875675 (+ 257us) server_negotiation.cc:407] Connection header received
0926 09:22:11.875761 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.875762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.875771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.875831 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.876208 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.876748 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.877529 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.877748 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.877834 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.878890 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.878892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.878892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.878899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.880382 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.880724 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.880726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.880727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.880790 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.880994 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.880995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.880996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.881153 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.881234 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.881406 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.881490 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:11.881603 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.875345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.24.235.67:52989)
0926 09:22:11.875608 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.875622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.875702 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.875846 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.875846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.876066 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:11.876141 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.876143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.876852 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.876853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.877444 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.877446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.877836 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.877837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.877913 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.878463 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:11.878467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:11.878777 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:11.880451 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.880452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.880453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.880634 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.880860 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:11.880861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:11.880861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:11.880920 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:11.881225 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:11.881226 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:11.881291 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.881379 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.881492 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:22:11 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:11.901443 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.893763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.24.235.126:43675)
0926 09:22:11.893839 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.893841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.893849 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:11.893873 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.893874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.893883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.893950 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:11.894541 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.895039 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.895792 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.895989 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.896103 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.898610 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:11.898612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:11.898613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:11.898621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:11.900142 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.900553 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.900555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.900555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.900620 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:11.900835 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:11.900836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:11.900837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:11.900961 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:11.901120 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.901246 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.901335 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:22:11.903322 27244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:11.904968 27244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:11.912390 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.907737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.24.235.67:44157)
0926 09:22:11.908441 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.908446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.908455 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:11.908492 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.908494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.908510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.908804 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:11.908885 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.909916 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.911311 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.911877 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.911991 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912053 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.912246 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.912287 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":85,"threads_started":1}
I20260926 09:22:11.912381 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.907655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:44157 (local address 127.0.0.1:40674)
0926 09:22:11.907850 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.907896 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.907969 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.908687 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.908688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.908690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:11.908735 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.908737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.910006 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.910007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.911205 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.911208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.912020 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.912021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.912112 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912126 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.912217 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.912276 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":29,"thread_start_us":60,"threads_started":1}
I20260926 09:22:11.912626 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.907685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38103 (local address 127.0.0.1:38642)
0926 09:22:11.907949 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.907959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.908002 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.908294 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.908297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.908299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:11.908372 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.908374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.909743 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.909744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.911538 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.911540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.912369 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.912370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.912432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912436 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.912487 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.912545 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":131,"thread_start_us":91,"threads_started":1}
I20260926 09:22:11.912828 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.907859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:43455 (local address 127.0.0.1:38262)
0926 09:22:11.908000 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:11.908013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:11.908071 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:11.908563 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:11.908564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:11.908566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:11.908613 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.908614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.909751 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.909752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.911044 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:11.911047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.912603 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.912604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:11.912665 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912669 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:11.912719 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:22:11.912759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":132,"thread_start_us":97,"threads_started":1}
I20260926 09:22:11.912927 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.907796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.24.235.65:38103)
0926 09:22:11.908126 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.908130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.908138 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:11.908170 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.908171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.908193 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.908283 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:11.908460 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.909648 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.911631 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.912278 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.912368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912419 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.912773 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.912821 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":79,"threads_started":1}
I20260926 09:22:11.913189 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 (DEFAULT_TABLE table=test-table [id=546d17321d814b3493beabd21de86118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:11.913340 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:11.908119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.24.235.66:43455)
0926 09:22:11.908295 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:11.908300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:11.908309 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:11.908353 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:11.908355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:11.908366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:11.908695 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:11.908756 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.909650 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.911131 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:11.911892 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:11.911991 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:11.912048 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:11.912985 (+ 937us) server_negotiation.cc:299] Negotiation successful
0926 09:22:11.913072 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260926 09:22:11.913630 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2baa5227a44b4da0894485ff2aa9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:11.913602 27381 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 (DEFAULT_TABLE table=test-table [id=546d17321d814b3493beabd21de86118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:11.913931 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2baa5227a44b4da0894485ff2aa9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:11.913908 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 (DEFAULT_TABLE table=test-table [id=546d17321d814b3493beabd21de86118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:11.914214 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2baa5227a44b4da0894485ff2aa9b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:11.916237 27750 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Bootstrap starting.
I20260926 09:22:11.917107 27750 tablet_bootstrap.cc:654] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:11.917484 27750 log.cc:824] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:11.917918 27753 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Bootstrap starting.
I20260926 09:22:11.918371 27750 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: No bootstrap required, opened a new log
I20260926 09:22:11.918443 27750 ts_tablet_manager.cc:1397] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:22:11.918596 27753 tablet_bootstrap.cc:654] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:11.918885 27753 log.cc:824] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:11.919361 27752 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Bootstrap starting.
I20260926 09:22:11.919785 27753 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: No bootstrap required, opened a new log
I20260926 09:22:11.919857 27753 ts_tablet_manager.cc:1397] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:11.919914 27752 tablet_bootstrap.cc:654] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:11.920272 27752 log.cc:824] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:11.920549 27750 raft_consensus.cc:348] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.920733 27750 raft_consensus.cc:374] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:11.920782 27750 raft_consensus.cc:729] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd8480b86f04a27a7563da34d1f9a69, State: Initialized, Role: FOLLOWER
I20260926 09:22:11.920907 27750 consensus_queue.cc:260] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.921223 27752 tablet_bootstrap.cc:492] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: No bootstrap required, opened a new log
I20260926 09:22:11.921244 27750 ts_tablet_manager.cc:1428] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 09:22:11.921309 27752 ts_tablet_manager.cc:1397] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:11.921402 27727 heartbeater.cc:503] Master 127.24.235.126:43675 was elected leader, sending a full tablet report...
I20260926 09:22:11.921828 27753 raft_consensus.cc:348] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.921939 27753 raft_consensus.cc:374] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:11.921991 27753 raft_consensus.cc:729] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Initialized, Role: FOLLOWER
I20260926 09:22:11.922135 27753 consensus_queue.cc:260] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.922411 27753 ts_tablet_manager.cc:1428] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:11.922431 27447 heartbeater.cc:503] Master 127.24.235.126:43675 was elected leader, sending a full tablet report...
I20260926 09:22:11.923408 27752 raft_consensus.cc:348] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.923580 27752 raft_consensus.cc:374] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:11.923624 27752 raft_consensus.cc:729] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Initialized, Role: FOLLOWER
I20260926 09:22:11.923743 27752 consensus_queue.cc:260] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:11.924005 27752 ts_tablet_manager.cc:1428] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:11.924058 27587 heartbeater.cc:503] Master 127.24.235.126:43675 was elected leader, sending a full tablet report...
W20260926 09:22:11.965135 27588 tablet.cc:2369] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:22:12.045921 27448 tablet.cc:2369] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:22:12.121248 27728 tablet.cc:2369] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:12.162710 27758 raft_consensus.cc:482] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:12.162844 27758 raft_consensus.cc:504] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:12.163261 27758 leader_election.cc:288] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157), c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103)
I20260926 09:22:12.166260 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.163423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:44157 (local address 127.24.235.66:54181)
0926 09:22:12.163493 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.163509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.163565 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.163766 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.163767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.163769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:12.163850 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.163852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.164519 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.164520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.165369 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.165372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.165928 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.166023 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.166025 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.166096 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.166135 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":48}
I20260926 09:22:12.166219 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.163507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:54181 (local address 127.24.235.67:44157)
0926 09:22:12.163551 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.163553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.163565 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:12.163657 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.163658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.163669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.163761 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:12.163933 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.164429 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165467 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.165843 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165931 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.165941 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.166102 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.166144 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:12.166267 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.163626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:57071 (local address 127.24.235.65:38103)
0926 09:22:12.163680 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.163682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.163798 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:22:12.163830 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.163832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.163841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.163917 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:12.164093 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.164556 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165532 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.165893 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165975 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.165983 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.166156 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.166198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:22:12.166277 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.163453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38103 (local address 127.24.235.66:57071)
0926 09:22:12.163698 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.163707 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.163773 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.163921 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.163922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.163923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:12.164015 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.164017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.164653 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.164654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.165440 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.165442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.165975 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.165976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.166058 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.166061 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.166135 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.166176 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":146,"threads_started":1}
I20260926 09:22:12.166628 27401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2baa5227a44b4da0894485ff2aa9b8" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5922fa6d5e24027a5910e32ae038c4d" is_pre_election: true
I20260926 09:22:12.166630 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2baa5227a44b4da0894485ff2aa9b8" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69" is_pre_election: true
I20260926 09:22:12.166796 27401 raft_consensus.cc:2454] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 0.
I20260926 09:22:12.166797 27681 raft_consensus.cc:2454] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 0.
I20260926 09:22:12.167048 27474 leader_election.cc:302] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:12.167196 27758 raft_consensus.cc:2781] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:12.167256 27758 raft_consensus.cc:482] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:12.167274 27758 raft_consensus.cc:3037] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.168030 27758 raft_consensus.cc:504] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:12.168176 27758 leader_election.cc:288] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 election: Requested vote from peers dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157), c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103)
I20260926 09:22:12.168337 27401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2baa5227a44b4da0894485ff2aa9b8" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5922fa6d5e24027a5910e32ae038c4d"
I20260926 09:22:12.168371 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2baa5227a44b4da0894485ff2aa9b8" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69"
I20260926 09:22:12.168437 27401 raft_consensus.cc:3037] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.168447 27681 raft_consensus.cc:3037] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.169207 27401 raft_consensus.cc:2454] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 1.
I20260926 09:22:12.169245 27681 raft_consensus.cc:2454] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 1.
I20260926 09:22:12.169792 27475 leader_election.cc:302] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcd8480b86f04a27a7563da34d1f9a69, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:12.169950 27758 raft_consensus.cc:2781] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:12.170122 27758 raft_consensus.cc:686] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 LEADER]: Becoming Leader. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Running, Role: LEADER
I20260926 09:22:12.170240 27758 consensus_queue.cc:237] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } }
I20260926 09:22:12.171154 27244 catalog_manager.cc:5495] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 reported cstate change: term changed from 0 to 1, leader changed from <none> to e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } health_report { overall_health: UNKNOWN } } }
Sep 26 09:22:12 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414531, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:12.225728 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.217970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43675 (local address 127.0.0.1:58456)
0926 09:22:12.218176 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.218186 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.218279 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.218456 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.218456 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.218802 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:12.218856 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.218859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.219521 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.219523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.220052 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.220054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.220429 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.220430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.220487 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.221139 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:12.221144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:12.222803 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:12.224439 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:12.224446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:12.224448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:12.224681 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:12.224926 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:12.224927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:12.224927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:12.224993 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:12.225352 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:12.225359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:12.225410 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.225484 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.225610 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":117,"threads_started":1}
I20260926 09:22:12.225739 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.218020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.24.235.126:43675)
0926 09:22:12.218101 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.218123 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.218257 (+ 134us) server_negotiation.cc:407] Connection header received
0926 09:22:12.218353 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.218354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.218365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.218439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:12.218955 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.219440 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.220156 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.220342 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.220427 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.222898 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:12.222900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:12.222900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:12.222924 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:12.224355 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:12.224803 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:12.224804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:12.224804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:12.224860 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:12.225119 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:12.225120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:12.225120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:12.225249 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:12.225310 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.225530 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.225658 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":46}
I20260926 09:22:12.228087 27244 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:58456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:22:12.228322 27244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:12.229974 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 4052232457e043fcb07e8a913a88fa1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=717aec16b4564e69adec0aa46972a9a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:12.229978 27381 tablet_service.cc:1467] Processing CreateTablet for tablet 4052232457e043fcb07e8a913a88fa1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=717aec16b4564e69adec0aa46972a9a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:12.230161 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4052232457e043fcb07e8a913a88fa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.230161 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4052232457e043fcb07e8a913a88fa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.230120 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 4052232457e043fcb07e8a913a88fa1a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=717aec16b4564e69adec0aa46972a9a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:12.230278 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4052232457e043fcb07e8a913a88fa1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.231455 27750 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: Bootstrap starting.
I20260926 09:22:12.231639 27753 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: Bootstrap starting.
I20260926 09:22:12.231652 27752 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: Bootstrap starting.
I20260926 09:22:12.232060 27752 tablet_bootstrap.cc:654] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.232067 27753 tablet_bootstrap.cc:654] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.232067 27750 tablet_bootstrap.cc:654] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.232689 27752 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: No bootstrap required, opened a new log
I20260926 09:22:12.232748 27752 ts_tablet_manager.cc:1397] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:12.232807 27750 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: No bootstrap required, opened a new log
I20260926 09:22:12.232859 27750 ts_tablet_manager.cc:1397] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:12.232900 27752 raft_consensus.cc:348] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.232962 27753 tablet_bootstrap.cc:492] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: No bootstrap required, opened a new log
I20260926 09:22:12.232985 27752 raft_consensus.cc:374] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.233036 27752 raft_consensus.cc:729] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.233053 27753 ts_tablet_manager.cc:1397] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:12.232995 27750 raft_consensus.cc:348] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.233085 27750 raft_consensus.cc:374] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.233114 27750 raft_consensus.cc:729] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd8480b86f04a27a7563da34d1f9a69, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.233088 27752 consensus_queue.cc:260] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.233170 27750 consensus_queue.cc:260] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.233202 27752 ts_tablet_manager.cc:1428] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:12.233223 27753 raft_consensus.cc:348] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.233268 27750 ts_tablet_manager.cc:1428] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:12.233302 27753 raft_consensus.cc:374] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.233330 27753 raft_consensus.cc:729] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.233337 27758 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.233377 27753 consensus_queue.cc:260] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.233410 27758 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.233480 27753 ts_tablet_manager.cc:1428] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:12.233520 27757 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.233640 27757 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.233631 27756 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.233723 27756 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.318786 27758 raft_consensus.cc:482] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:12.318895 27758 raft_consensus.cc:504] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.319154 27758 leader_election.cc:288] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:12.319469 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4052232457e043fcb07e8a913a88fa1a" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69" is_pre_election: true
I20260926 09:22:12.319469 27401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4052232457e043fcb07e8a913a88fa1a" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5922fa6d5e24027a5910e32ae038c4d" is_pre_election: true
I20260926 09:22:12.319600 27681 raft_consensus.cc:2454] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 0.
I20260926 09:22:12.319653 27401 raft_consensus.cc:2454] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 0.
I20260926 09:22:12.319864 27475 leader_election.cc:302] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcd8480b86f04a27a7563da34d1f9a69, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:12.319964 27758 raft_consensus.cc:2781] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:12.320008 27758 raft_consensus.cc:482] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:12.320052 27758 raft_consensus.cc:3037] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.320708 27758 raft_consensus.cc:504] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.320849 27758 leader_election.cc:288] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 election: Requested vote from peers c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:12.321069 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4052232457e043fcb07e8a913a88fa1a" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69"
I20260926 09:22:12.321053 27401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4052232457e043fcb07e8a913a88fa1a" candidate_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5922fa6d5e24027a5910e32ae038c4d"
I20260926 09:22:12.321147 27681 raft_consensus.cc:3037] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.321168 27401 raft_consensus.cc:3037] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.321786 27681 raft_consensus.cc:2454] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 1.
I20260926 09:22:12.321931 27401 raft_consensus.cc:2454] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16dd7f99f1a4c8f9f7976e847ab6b38 in term 1.
I20260926 09:22:12.322021 27475 leader_election.cc:302] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcd8480b86f04a27a7563da34d1f9a69, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:12.322141 27758 raft_consensus.cc:2781] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:12.322230 27758 raft_consensus.cc:686] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 LEADER]: Becoming Leader. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Running, Role: LEADER
I20260926 09:22:12.322279 27758 consensus_queue.cc:237] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.322600 27760 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: TxnStatusTablet state changed. Reason: New leader e16dd7f99f1a4c8f9f7976e847ab6b38. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.322680 27760 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:12.322855 27772 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:22:12.323022 27772 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:22:12.322983 27242 catalog_manager.cc:5495] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 reported cstate change: term changed from 0 to 1, leader changed from <none> to e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:12.329186 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.326510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.24.235.66:43455)
0926 09:22:12.326564 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.326566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.326576 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:12.326708 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.326709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.326726 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.326790 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.326946 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.327446 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.328266 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.328480 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.328569 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.328773 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.328881 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.328966 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.329068 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.329094 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:22:12.329186 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.326435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:43455 (local address 127.0.0.1:38266)
0926 09:22:12.326510 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.326522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.326576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.326803 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.326804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.326805 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:12.326858 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.326861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.327535 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.327536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.328157 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.328159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.328570 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.328571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.328649 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.328671 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.328955 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.328964 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.329000 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.329104 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:22:12.331418 27401 raft_consensus.cc:1261] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Refusing update from remote peer e16dd7f99f1a4c8f9f7976e847ab6b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.331418 27681 raft_consensus.cc:1261] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Refusing update from remote peer e16dd7f99f1a4c8f9f7976e847ab6b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.331951 27760 consensus_queue.cc:1035] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.332123 27758 consensus_queue.cc:1035] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.333461 27757 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: TxnStatusTablet state changed. Reason: New leader e16dd7f99f1a4c8f9f7976e847ab6b38. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.333477 27756 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: TxnStatusTablet state changed. Reason: New leader e16dd7f99f1a4c8f9f7976e847ab6b38. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.333580 27757 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.333587 27756 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.334653 27758 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.334762 27758 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:12.334834 27775 mvcc.cc:205] Tried to move back new op lower bound from 7333537924426608640 to 7333537924392669184. Current Snapshot: MvccSnapshot[applied={T|T < 7333537924426608640}]
I20260926 09:22:12.334841 27756 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.334908 27756 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P dcd8480b86f04a27a7563da34d1f9a69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.335083 27777 mvcc.cc:205] Tried to move back new op lower bound from 7333537924426608640 to 7333537924392669184. Current Snapshot: MvccSnapshot[applied={T|T < 7333537924426608640}]
I20260926 09:22:12.335999 27758 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.336118 27758 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P e16dd7f99f1a4c8f9f7976e847ab6b38: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:12.336361 27757 tablet_replica.cc:439] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } } }
I20260926 09:22:12.336458 27757 tablet_replica.cc:442] T 4052232457e043fcb07e8a913a88fa1a P c5922fa6d5e24027a5910e32ae038c4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:12.342554 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.338070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.24.235.66:43455)
0926 09:22:12.338132 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.338135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.340155 (+ 2020us) server_negotiation.cc:407] Connection header received
0926 09:22:12.340222 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.340223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.340235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.340284 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.340434 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.340915 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.341832 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.342043 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.342135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.342284 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.342318 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.342370 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.342434 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.342455 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:12.348253 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.345623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:50223 (local address 127.24.235.66:43455)
0926 09:22:12.345665 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.345667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.345718 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:22:12.345833 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.345834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.345844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.345907 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.346064 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.346532 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.347357 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.347553 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.347642 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.347919 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.347959 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.348035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.348103 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.348135 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:22:12.348611 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.345443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:43455 (local address 127.24.235.66:50223)
0926 09:22:12.345648 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.345661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.345749 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.345911 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.345911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.345913 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:12.345985 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.345987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.346625 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.346626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.347252 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.347255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.347664 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.347665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.347731 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.347755 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.348038 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.348040 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.348113 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.348157 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260926 09:22:12.351238 27681 raft_consensus.cc:1261] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Refusing update from remote peer e16dd7f99f1a4c8f9f7976e847ab6b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.351227 27401 raft_consensus.cc:1261] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Refusing update from remote peer e16dd7f99f1a4c8f9f7976e847ab6b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.351455 27782 consensus_queue.cc:1035] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.351585 27760 consensus_queue.cc:1035] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.364392 27242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:12.364519 27242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:12.365348 27242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Sending DeleteTablet for 3 replicas of tablet 5f2baa5227a44b4da0894485ff2aa9b8
I20260926 09:22:12.365775 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:12 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38262
I20260926 09:22:12.365785 27381 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:12 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38642
I20260926 09:22:12.366070 27791 tablet_replica.cc:330] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: stopping tablet replica
I20260926 09:22:12.366073 27661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2baa5227a44b4da0894485ff2aa9b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:12 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:40674
I20260926 09:22:12.366182 27791 raft_consensus.cc:2229] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:12.366133 27242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:12.366267 27242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:12.366277 27791 raft_consensus.cc:2258] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:12.366768 27792 tablet_replica.cc:330] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: stopping tablet replica
I20260926 09:22:12.366940 27792 raft_consensus.cc:2229] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:12.367444 27792 raft_consensus.cc:2258] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:12.367499 27521 tablet_service.cc:1467] Processing CreateTablet for tablet dfc39b0f4eb1470db2cea30915216663 (DEFAULT_TABLE table=test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:12.367497 27381 tablet_service.cc:1467] Processing CreateTablet for tablet dfc39b0f4eb1470db2cea30915216663 (DEFAULT_TABLE table=test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:12.367631 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc39b0f4eb1470db2cea30915216663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.367632 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc39b0f4eb1470db2cea30915216663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.367817 27793 tablet_replica.cc:330] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: stopping tablet replica
I20260926 09:22:12.367962 27793 raft_consensus.cc:2229] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:12.368053 27793 raft_consensus.cc:2258] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:12.368582 27793 ts_tablet_manager.cc:1905] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:12.369190 27753 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Bootstrap starting.
I20260926 09:22:12.369326 27792 ts_tablet_manager.cc:1905] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:12.369882 27753 tablet_bootstrap.cc:654] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.370347 27661 tablet_service.cc:1467] Processing CreateTablet for tablet dfc39b0f4eb1470db2cea30915216663 (DEFAULT_TABLE table=test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:12.370445 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc39b0f4eb1470db2cea30915216663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:12.370507 27753 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: No bootstrap required, opened a new log
I20260926 09:22:12.370561 27753 ts_tablet_manager.cc:1397] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:12.370710 27753 raft_consensus.cc:348] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.370787 27753 raft_consensus.cc:374] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.370811 27753 raft_consensus.cc:729] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.370855 27753 consensus_queue.cc:260] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.370934 27753 ts_tablet_manager.cc:1428] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:12.371181 27793 ts_tablet_manager.cc:1918] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:12.371249 27793 log.cc:1196] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/5f2baa5227a44b4da0894485ff2aa9b8
I20260926 09:22:12.371520 27793 ts_tablet_manager.cc:1939] T 5f2baa5227a44b4da0894485ff2aa9b8 P dcd8480b86f04a27a7563da34d1f9a69: Deleting consensus metadata
I20260926 09:22:12.371716 27750 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Bootstrap starting.
I20260926 09:22:12.372090 27750 tablet_bootstrap.cc:654] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.372500 27231 catalog_manager.cc:4841] TS dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157): tablet 5f2baa5227a44b4da0894485ff2aa9b8 (table test-table [id=546d17321d814b3493beabd21de86118]) successfully deleted
I20260926 09:22:12.372756 27750 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: No bootstrap required, opened a new log
I20260926 09:22:12.372795 27750 ts_tablet_manager.cc:1397] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:12.372941 27750 raft_consensus.cc:348] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.373039 27750 raft_consensus.cc:374] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.373062 27750 raft_consensus.cc:729] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd8480b86f04a27a7563da34d1f9a69, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.373106 27750 consensus_queue.cc:260] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.373193 27750 ts_tablet_manager.cc:1428] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:12.373610 27752 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Bootstrap starting.
I20260926 09:22:12.373710 27791 ts_tablet_manager.cc:1905] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:12.374029 27752 tablet_bootstrap.cc:654] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:12.374045 27792 ts_tablet_manager.cc:1918] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:12.374145 27792 log.cc:1196] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/5f2baa5227a44b4da0894485ff2aa9b8
I20260926 09:22:12.374459 27792 ts_tablet_manager.cc:1939] T 5f2baa5227a44b4da0894485ff2aa9b8 P c5922fa6d5e24027a5910e32ae038c4d: Deleting consensus metadata
I20260926 09:22:12.374837 27752 tablet_bootstrap.cc:492] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: No bootstrap required, opened a new log
I20260926 09:22:12.374888 27752 ts_tablet_manager.cc:1397] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:12.374964 27230 catalog_manager.cc:4841] TS c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103): tablet 5f2baa5227a44b4da0894485ff2aa9b8 (table test-table [id=546d17321d814b3493beabd21de86118]) successfully deleted
I20260926 09:22:12.375020 27752 raft_consensus.cc:348] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.375095 27752 raft_consensus.cc:374] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:12.375115 27752 raft_consensus.cc:729] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Initialized, Role: FOLLOWER
I20260926 09:22:12.375156 27752 consensus_queue.cc:260] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.375254 27752 ts_tablet_manager.cc:1428] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:12.375664 27791 ts_tablet_manager.cc:1918] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:12.375733 27791 log.cc:1196] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/5f2baa5227a44b4da0894485ff2aa9b8
I20260926 09:22:12.376037 27791 ts_tablet_manager.cc:1939] T 5f2baa5227a44b4da0894485ff2aa9b8 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting consensus metadata
I20260926 09:22:12.376398 27231 catalog_manager.cc:4841] TS e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455): tablet 5f2baa5227a44b4da0894485ff2aa9b8 (table test-table [id=546d17321d814b3493beabd21de86118]) successfully deleted
I20260926 09:22:12.420171 27757 raft_consensus.cc:482] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:12.420286 27757 raft_consensus.cc:504] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.420729 27757 leader_election.cc:288] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:12.424618 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.421045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:36783 (local address 127.24.235.66:43455)
0926 09:22:12.421106 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.421108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.421193 (+ 85us) server_negotiation.cc:407] Connection header received
0926 09:22:12.421312 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.421313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.421324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.421362 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:12.421646 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.422432 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.423782 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.424194 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.424286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.424297 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.424496 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.424545 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:22:12.424656 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.420941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:43455 (local address 127.24.235.65:36783)
0926 09:22:12.421124 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.421135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.421225 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.421435 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.421436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.421438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:12.421542 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.421545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.422526 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.422527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.423663 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.423666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.424285 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.424286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.424369 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.424373 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.424477 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.424535 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":4,"thread_start_us":125,"threads_started":1}
I20260926 09:22:12.424733 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.421266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:52573 (local address 127.24.235.67:44157)
0926 09:22:12.421309 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.421311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.421391 (+ 80us) server_negotiation.cc:407] Connection header received
0926 09:22:12.421463 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.421464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.421474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.421555 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:12.421689 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.422232 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.423759 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.424335 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.424422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.424432 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.424598 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.424656 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:12.424929 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.421186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:44157 (local address 127.24.235.65:52573)
0926 09:22:12.421338 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.421347 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.421395 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.421566 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.421567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.421568 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:12.421627 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.421628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.422317 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.422318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.423663 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.423666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.424425 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.424425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.424510 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.424513 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.424813 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.424859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260926 09:22:12.425094 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc39b0f4eb1470db2cea30915216663" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" is_pre_election: true
I20260926 09:22:12.425244 27541 raft_consensus.cc:2454] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:12.425256 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc39b0f4eb1470db2cea30915216663" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69" is_pre_election: true
I20260926 09:22:12.425340 27681 raft_consensus.cc:2454] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:12.425474 27335 leader_election.cc:302] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:12.425698 27757 raft_consensus.cc:2781] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:12.425745 27757 raft_consensus.cc:482] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:12.425765 27757 raft_consensus.cc:3037] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.426450 27757 raft_consensus.cc:504] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.426625 27757 leader_election.cc:288] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Requested vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:12.426792 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc39b0f4eb1470db2cea30915216663" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
I20260926 09:22:12.426872 27541 raft_consensus.cc:3037] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.426827 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc39b0f4eb1470db2cea30915216663" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69"
I20260926 09:22:12.426906 27681 raft_consensus.cc:3037] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:12.427569 27681 raft_consensus.cc:2454] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:12.427764 27335 leader_election.cc:302] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, dcd8480b86f04a27a7563da34d1f9a69; no voters:
I20260926 09:22:12.427775 27541 raft_consensus.cc:2454] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:12.427855 27757 raft_consensus.cc:2781] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:12.428057 27757 raft_consensus.cc:686] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 1 LEADER]: Becoming Leader. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Running, Role: LEADER
I20260926 09:22:12.428143 27757 consensus_queue.cc:237] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:12.428876 27242 catalog_manager.cc:5495] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65). New cstate: current_term: 1 leader_uuid: "c5922fa6d5e24027a5910e32ae038c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:12.449004 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.446118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.24.235.65:38103)
0926 09:22:12.446155 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.446157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.446166 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:12.446252 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.446252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.446291 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.446366 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.446507 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.446983 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.448046 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.448265 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.448322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.448635 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.448742 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.448801 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.448923 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.448942 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:22:12.454588 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.451923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:59551 (local address 127.24.235.66:43455)
0926 09:22:12.451976 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.451978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.452040 (+ 62us) server_negotiation.cc:407] Connection header received
0926 09:22:12.452093 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.452094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.452104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.452170 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.452266 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.452976 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.453755 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.454074 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.454156 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.454309 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.454345 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.454400 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.454478 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.454504 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:22:12.454651 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.451833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:43455 (local address 127.24.235.65:59551)
0926 09:22:12.451976 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.451988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.452049 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.452174 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.452174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.452175 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:12.452226 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.452228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.453105 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.453105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.453665 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.453666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.454163 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.454163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.454213 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.454226 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.454425 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.454427 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.454480 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.454516 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260926 09:22:12.457477 27681 raft_consensus.cc:1261] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.457558 27541 raft_consensus.cc:1261] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:12.457895 27799 consensus_queue.cc:1035] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.458156 27757 consensus_queue.cc:1035] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:12.459427 27776 mvcc.cc:205] Tried to move back new op lower bound from 7333537924943884288 to 7333537924826218496. Current Snapshot: MvccSnapshot[applied={T|T < 7333537924943884288}]
I20260926 09:22:12.466248 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.463620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:56705 (local address 127.24.235.65:38103)
0926 09:22:12.463657 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:12.463659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:12.463692 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:22:12.463767 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:12.463767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:12.463779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:12.463827 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:12.463993 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.464717 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.465511 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.465720 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.465804 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.465972 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.466007 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.466053 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:12.466146 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 09:22:12.466170 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:12.466271 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:12.463599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38103 (local address 127.24.235.66:56705)
0926 09:22:12.463650 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:12.463662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:12.463707 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:12.463869 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:12.463870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:12.463871 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:12.463916 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.463918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.464806 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.464807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.465429 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:12.465432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:12.465805 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:12.465805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:12.465884 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:12.465890 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:12.466087 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:12.466090 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:12.466141 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:22:12.466196 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:22:12.466619 27776 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:22:13.988193 27242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:13.988319 27242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:13.989321 27242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Sending DeleteTablet for 3 replicas of tablet dfc39b0f4eb1470db2cea30915216663
I20260926 09:22:13.989638 27381 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc39b0f4eb1470db2cea30915216663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:13 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38642
I20260926 09:22:13.989638 27661 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc39b0f4eb1470db2cea30915216663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:13 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:40674
I20260926 09:22:13.989871 27830 tablet_replica.cc:330] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: stopping tablet replica
I20260926 09:22:13.989925 27830 raft_consensus.cc:2229] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:13.990010 27830 raft_consensus.cc:2258] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:13.990252 27830 ts_tablet_manager.cc:1905] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:13.991291 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc39b0f4eb1470db2cea30915216663 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:13 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38262
I20260926 09:22:13.992235 27829 tablet_replica.cc:330] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: stopping tablet replica
I20260926 09:22:13.992231 27830 ts_tablet_manager.cc:1918] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:22:13.992296 27830 log.cc:1196] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/dfc39b0f4eb1470db2cea30915216663
I20260926 09:22:13.992306 27829 raft_consensus.cc:2229] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:13.992348 27829 raft_consensus.cc:2258] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:13.992462 27832 tablet_replica.cc:330] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: stopping tablet replica
I20260926 09:22:13.992525 27832 raft_consensus.cc:2229] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:13.992579 27832 raft_consensus.cc:2258] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:13.992669 27830 ts_tablet_manager.cc:1939] T dfc39b0f4eb1470db2cea30915216663 P c5922fa6d5e24027a5910e32ae038c4d: Deleting consensus metadata
I20260926 09:22:13.992828 27832 ts_tablet_manager.cc:1905] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:13.992990 27230 catalog_manager.cc:4841] TS c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103): tablet dfc39b0f4eb1470db2cea30915216663 (table test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]) successfully deleted
I20260926 09:22:13.993218 27829 ts_tablet_manager.cc:1905] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:13.995000 27832 ts_tablet_manager.cc:1918] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:22:13.995079 27832 log.cc:1196] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/dfc39b0f4eb1470db2cea30915216663
I20260926 09:22:13.995203 27829 ts_tablet_manager.cc:1918] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:22:13.995270 27829 log.cc:1196] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/dfc39b0f4eb1470db2cea30915216663
I20260926 09:22:13.995332 27832 ts_tablet_manager.cc:1939] T dfc39b0f4eb1470db2cea30915216663 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting consensus metadata
I20260926 09:22:13.995505 27829 ts_tablet_manager.cc:1939] T dfc39b0f4eb1470db2cea30915216663 P dcd8480b86f04a27a7563da34d1f9a69: Deleting consensus metadata
I20260926 09:22:13.995733 27231 catalog_manager.cc:4841] TS e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455): tablet dfc39b0f4eb1470db2cea30915216663 (table test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]) successfully deleted
I20260926 09:22:13.996064 27231 catalog_manager.cc:4841] TS dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157): tablet dfc39b0f4eb1470db2cea30915216663 (table test-table [id=1aa9e9575d254ccb9d8a0de1ad4b543e]) successfully deleted
Sep 26 09:22:14 dist-test-slave-7zlw krb5kdc[27199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:22:14 dist-test-slave-7zlw krb5kdc[27199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:14.020088 27839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:14.011496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.24.235.126:43675)
0926 09:22:14.011667 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:14.011670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:14.011696 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:22:14.011734 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:14.011734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:14.011744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:14.011813 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:14.012511 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:14.013118 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:14.013871 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:14.014082 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:14.014160 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:14.017079 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:14.017081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:14.017082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:14.017091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:14.018939 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:14.019276 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:14.019276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:14.019277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:14.019336 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:14.019550 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:14.019551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:14.019551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:14.019687 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:14.019771 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:14.019910 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 09:22:14.019982 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260926 09:22:14.022107 27242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:14.022300 27242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:14.023885 27381 tablet_service.cc:1467] Processing CreateTablet for tablet 60f293932be842258a53505da5e44197 (DEFAULT_TABLE table=test-table [id=7f785c7ed48447018971d2a25e8db2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.023938 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 60f293932be842258a53505da5e44197 (DEFAULT_TABLE table=test-table [id=7f785c7ed48447018971d2a25e8db2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.024019 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f293932be842258a53505da5e44197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.024123 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f293932be842258a53505da5e44197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.025120 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 60f293932be842258a53505da5e44197 (DEFAULT_TABLE table=test-table [id=7f785c7ed48447018971d2a25e8db2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.025246 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f293932be842258a53505da5e44197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.025753 27840 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Bootstrap starting.
I20260926 09:22:14.025751 27841 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Bootstrap starting.
I20260926 09:22:14.026177 27841 tablet_bootstrap.cc:654] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.026479 27840 tablet_bootstrap.cc:654] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.026726 27841 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: No bootstrap required, opened a new log
I20260926 09:22:14.026758 27841 ts_tablet_manager.cc:1397] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:14.026911 27841 raft_consensus.cc:348] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.027025 27841 raft_consensus.cc:374] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.027055 27841 raft_consensus.cc:729] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd8480b86f04a27a7563da34d1f9a69, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.027138 27841 consensus_queue.cc:260] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.027189 27840 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: No bootstrap required, opened a new log
I20260926 09:22:14.027221 27844 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Bootstrap starting.
I20260926 09:22:14.027231 27840 ts_tablet_manager.cc:1397] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:14.027312 27841 ts_tablet_manager.cc:1428] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:14.027343 27840 raft_consensus.cc:348] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.027415 27840 raft_consensus.cc:374] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.027436 27840 raft_consensus.cc:729] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.027499 27840 consensus_queue.cc:260] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.027701 27840 ts_tablet_manager.cc:1428] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:14.027801 27844 tablet_bootstrap.cc:654] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.028470 27844 tablet_bootstrap.cc:492] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: No bootstrap required, opened a new log
I20260926 09:22:14.028522 27844 ts_tablet_manager.cc:1397] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:14.028638 27844 raft_consensus.cc:348] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.028715 27844 raft_consensus.cc:374] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.028748 27844 raft_consensus.cc:729] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.028805 27844 consensus_queue.cc:260] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.028896 27844 ts_tablet_manager.cc:1428] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:14.219251 27824 raft_consensus.cc:482] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:14.219360 27824 raft_consensus.cc:504] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.219501 27824 leader_election.cc:288] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:14.219973 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f293932be842258a53505da5e44197" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" is_pre_election: true
I20260926 09:22:14.219977 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f293932be842258a53505da5e44197" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69" is_pre_election: true
I20260926 09:22:14.220088 27681 raft_consensus.cc:2454] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:14.220115 27541 raft_consensus.cc:2454] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:14.220355 27335 leader_election.cc:302] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:14.220482 27824 raft_consensus.cc:2781] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:14.220532 27824 raft_consensus.cc:482] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:14.220561 27824 raft_consensus.cc:3037] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.221370 27824 raft_consensus.cc:504] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.221521 27824 leader_election.cc:288] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Requested vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:14.221686 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f293932be842258a53505da5e44197" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
I20260926 09:22:14.221784 27541 raft_consensus.cc:3037] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.221774 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f293932be842258a53505da5e44197" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69"
I20260926 09:22:14.221851 27681 raft_consensus.cc:3037] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.222785 27681 raft_consensus.cc:2454] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:14.222862 27541 raft_consensus.cc:2454] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:14.222996 27335 leader_election.cc:302] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, dcd8480b86f04a27a7563da34d1f9a69; no voters:
I20260926 09:22:14.223114 27824 raft_consensus.cc:2781] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:14.223171 27824 raft_consensus.cc:686] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 1 LEADER]: Becoming Leader. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Running, Role: LEADER
I20260926 09:22:14.223232 27824 consensus_queue.cc:237] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.223985 27244 catalog_manager.cc:5495] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65). New cstate: current_term: 1 leader_uuid: "c5922fa6d5e24027a5910e32ae038c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:14.258235 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:14.255585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.24.235.65:38103)
0926 09:22:14.255771 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:14.255772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:14.255779 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:22:14.255799 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:14.255799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:14.255809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:14.255867 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:14.256050 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:14.256569 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:14.257494 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:14.257698 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:14.257748 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:14.257983 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:14.258009 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:14.258047 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:14.258106 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 09:22:14.258125 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260926 09:22:14.260824 27541 raft_consensus.cc:1261] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:14.260824 27681 raft_consensus.cc:1261] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:14.261150 27824 consensus_queue.cc:1035] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:14.261263 27820 consensus_queue.cc:1035] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:14.271360 27242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:14.271488 27242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:14.272197 27242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Sending DeleteTablet for 3 replicas of tablet 60f293932be842258a53505da5e44197
I20260926 09:22:14.272508 27381 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f293932be842258a53505da5e44197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:14 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38642
I20260926 09:22:14.272657 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f293932be842258a53505da5e44197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:14 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38262
I20260926 09:22:14.272668 27830 tablet_replica.cc:330] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: stopping tablet replica
I20260926 09:22:14.272683 27661 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f293932be842258a53505da5e44197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:14 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:40674
I20260926 09:22:14.272770 27832 tablet_replica.cc:330] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: stopping tablet replica
I20260926 09:22:14.272768 27830 raft_consensus.cc:2229] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:14.272785 27829 tablet_replica.cc:330] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: stopping tablet replica
I20260926 09:22:14.272835 27829 raft_consensus.cc:2229] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:14.272840 27830 raft_consensus.cc:2258] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:14.272838 27832 raft_consensus.cc:2229] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:14.272891 27832 raft_consensus.cc:2258] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:14.272886 27829 raft_consensus.cc:2258] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:14.273193 27829 ts_tablet_manager.cc:1905] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:14.273469 27830 ts_tablet_manager.cc:1905] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:14.275009 27829 ts_tablet_manager.cc:1918] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:14.275074 27829 log.cc:1196] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/60f293932be842258a53505da5e44197
I20260926 09:22:14.275125 27832 ts_tablet_manager.cc:1905] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:14.274991 27242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:14.275259 27242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:14.275339 27829 ts_tablet_manager.cc:1939] T 60f293932be842258a53505da5e44197 P dcd8480b86f04a27a7563da34d1f9a69: Deleting consensus metadata
I20260926 09:22:14.275696 27231 catalog_manager.cc:4841] TS dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157): tablet 60f293932be842258a53505da5e44197 (table test-table [id=7f785c7ed48447018971d2a25e8db2b7]) successfully deleted
I20260926 09:22:14.276603 27830 ts_tablet_manager.cc:1918] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:14.276664 27830 log.cc:1196] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/60f293932be842258a53505da5e44197
I20260926 09:22:14.276877 27830 ts_tablet_manager.cc:1939] T 60f293932be842258a53505da5e44197 P c5922fa6d5e24027a5910e32ae038c4d: Deleting consensus metadata
I20260926 09:22:14.277208 27832 ts_tablet_manager.cc:1918] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:14.277209 27230 catalog_manager.cc:4841] TS c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103): tablet 60f293932be842258a53505da5e44197 (table test-table [id=7f785c7ed48447018971d2a25e8db2b7]) successfully deleted
I20260926 09:22:14.277289 27832 log.cc:1196] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/60f293932be842258a53505da5e44197
I20260926 09:22:14.277560 27832 ts_tablet_manager.cc:1939] T 60f293932be842258a53505da5e44197 P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting consensus metadata
I20260926 09:22:14.277585 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 922b6c5979334369a3d0e071b1ad65ca (DEFAULT_TABLE table=test-table [id=f16b63aa53804ebab4d3b16007262ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.277693 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922b6c5979334369a3d0e071b1ad65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.277849 27231 catalog_manager.cc:4841] TS e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455): tablet 60f293932be842258a53505da5e44197 (table test-table [id=7f785c7ed48447018971d2a25e8db2b7]) successfully deleted
I20260926 09:22:14.278734 27841 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Bootstrap starting.
I20260926 09:22:14.278772 27381 tablet_service.cc:1467] Processing CreateTablet for tablet 922b6c5979334369a3d0e071b1ad65ca (DEFAULT_TABLE table=test-table [id=f16b63aa53804ebab4d3b16007262ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.278867 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922b6c5979334369a3d0e071b1ad65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.279561 27841 tablet_bootstrap.cc:654] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.279963 27840 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d: Bootstrap starting.
I20260926 09:22:14.280069 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 922b6c5979334369a3d0e071b1ad65ca (DEFAULT_TABLE table=test-table [id=f16b63aa53804ebab4d3b16007262ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:14.280169 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 922b6c5979334369a3d0e071b1ad65ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:14.280313 27840 tablet_bootstrap.cc:654] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.281239 27841 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: No bootstrap required, opened a new log
I20260926 09:22:14.281292 27841 ts_tablet_manager.cc:1397] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:22:14.281440 27841 raft_consensus.cc:348] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.281518 27841 raft_consensus.cc:374] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.281517 27844 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Bootstrap starting.
I20260926 09:22:14.281540 27841 raft_consensus.cc:729] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd8480b86f04a27a7563da34d1f9a69, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.281582 27841 consensus_queue.cc:260] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.281673 27841 ts_tablet_manager.cc:1428] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:14.282019 27840 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d: No bootstrap required, opened a new log
I20260926 09:22:14.282090 27840 ts_tablet_manager.cc:1397] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:14.282214 27844 tablet_bootstrap.cc:654] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:14.282217 27840 raft_consensus.cc:348] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.282303 27840 raft_consensus.cc:374] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.282323 27840 raft_consensus.cc:729] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.282368 27840 consensus_queue.cc:260] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.282461 27840 ts_tablet_manager.cc:1428] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:14.283052 27844 tablet_bootstrap.cc:492] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: No bootstrap required, opened a new log
I20260926 09:22:14.283103 27844 ts_tablet_manager.cc:1397] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:14.283221 27844 raft_consensus.cc:348] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.283293 27844 raft_consensus.cc:374] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:14.283313 27844 raft_consensus.cc:729] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16dd7f99f1a4c8f9f7976e847ab6b38, State: Initialized, Role: FOLLOWER
I20260926 09:22:14.283362 27844 consensus_queue.cc:260] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.283478 27844 ts_tablet_manager.cc:1428] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:14.303617 27824 raft_consensus.cc:482] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:14.303717 27824 raft_consensus.cc:504] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.303871 27824 leader_election.cc:288] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:14.304061 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b6c5979334369a3d0e071b1ad65ca" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" is_pre_election: true
I20260926 09:22:14.304167 27541 raft_consensus.cc:2454] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:14.304101 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b6c5979334369a3d0e071b1ad65ca" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69" is_pre_election: true
I20260926 09:22:14.304224 27681 raft_consensus.cc:2454] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 0.
I20260926 09:22:14.304329 27335 leader_election.cc:302] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:14.304455 27824 raft_consensus.cc:2781] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:14.304579 27824 raft_consensus.cc:482] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:14.304622 27824 raft_consensus.cc:3037] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.305567 27824 raft_consensus.cc:504] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.305697 27824 leader_election.cc:288] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Requested vote from peers e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455), dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157)
I20260926 09:22:14.305876 27541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b6c5979334369a3d0e071b1ad65ca" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38"
I20260926 09:22:14.305917 27681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b6c5979334369a3d0e071b1ad65ca" candidate_uuid: "c5922fa6d5e24027a5910e32ae038c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd8480b86f04a27a7563da34d1f9a69"
I20260926 09:22:14.305992 27541 raft_consensus.cc:3037] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.306008 27681 raft_consensus.cc:3037] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:14.306605 27541 raft_consensus.cc:2454] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:14.306663 27681 raft_consensus.cc:2454] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5922fa6d5e24027a5910e32ae038c4d in term 1.
I20260926 09:22:14.306788 27335 leader_election.cc:302] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5922fa6d5e24027a5910e32ae038c4d, e16dd7f99f1a4c8f9f7976e847ab6b38; no voters:
I20260926 09:22:14.306962 27824 raft_consensus.cc:2781] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:14.307019 27824 raft_consensus.cc:686] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [term 1 LEADER]: Becoming Leader. State: Replica: c5922fa6d5e24027a5910e32ae038c4d, State: Running, Role: LEADER
I20260926 09:22:14.307101 27824 consensus_queue.cc:237] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } }
I20260926 09:22:14.307724 27244 catalog_manager.cc:5495] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65). New cstate: current_term: 1 leader_uuid: "c5922fa6d5e24027a5910e32ae038c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5922fa6d5e24027a5910e32ae038c4d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:14.315656 27681 raft_consensus.cc:1261] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:14.315709 27541 raft_consensus.cc:1261] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Refusing update from remote peer c5922fa6d5e24027a5910e32ae038c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:14.315925 27824 consensus_queue.cc:1035] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd8480b86f04a27a7563da34d1f9a69" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:14.316076 27820 consensus_queue.cc:1035] T 922b6c5979334369a3d0e071b1ad65ca P c5922fa6d5e24027a5910e32ae038c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16dd7f99f1a4c8f9f7976e847ab6b38" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:22:14.320384 27812 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 09:22:15.838384 27243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:15.838488 27243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:15.839449 27243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90bd66cdd56c4f60ad47c22443425f2e: Sending DeleteTablet for 3 replicas of tablet 922b6c5979334369a3d0e071b1ad65ca
I20260926 09:22:15.839754 27661 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b6c5979334369a3d0e071b1ad65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:15 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:40674
I20260926 09:22:15.839751 27381 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b6c5979334369a3d0e071b1ad65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:15 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38642
I20260926 09:22:15.839936 27521 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b6c5979334369a3d0e071b1ad65ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:15 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38262
I20260926 09:22:15.840123 27882 tablet_replica.cc:330] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: stopping tablet replica
I20260926 09:22:15.840116 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27316
I20260926 09:22:15.840188 27882 raft_consensus.cc:2229] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:15.840939 27882 raft_consensus.cc:2258] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:15.841120 27881 tablet_replica.cc:330] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: stopping tablet replica
I20260926 09:22:15.841179 27881 raft_consensus.cc:2229] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:15.841236 27881 raft_consensus.cc:2258] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:15.841423 27882 ts_tablet_manager.cc:1905] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:15.842808 27881 ts_tablet_manager.cc:1905] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:15.843178 27882 ts_tablet_manager.cc:1918] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:22:15.843235 27882 log.cc:1196] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/922b6c5979334369a3d0e071b1ad65ca
I20260926 09:22:15.843453 27882 ts_tablet_manager.cc:1939] T 922b6c5979334369a3d0e071b1ad65ca P e16dd7f99f1a4c8f9f7976e847ab6b38: Deleting consensus metadata
I20260926 09:22:15.845108 27881 ts_tablet_manager.cc:1918] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 09:22:15.845171 27881 log.cc:1196] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/922b6c5979334369a3d0e071b1ad65ca
I20260926 09:22:15.845337 27231 catalog_manager.cc:4841] TS e16dd7f99f1a4c8f9f7976e847ab6b38 (127.24.235.66:43455): tablet 922b6c5979334369a3d0e071b1ad65ca (table test-table [id=f16b63aa53804ebab4d3b16007262ad2]) successfully deleted
I20260926 09:22:15.845386 27881 ts_tablet_manager.cc:1939] T 922b6c5979334369a3d0e071b1ad65ca P dcd8480b86f04a27a7563da34d1f9a69: Deleting consensus metadata
I20260926 09:22:15.846029 27231 catalog_manager.cc:4841] TS dcd8480b86f04a27a7563da34d1f9a69 (127.24.235.67:44157): tablet 922b6c5979334369a3d0e071b1ad65ca (table test-table [id=f16b63aa53804ebab4d3b16007262ad2]) successfully deleted
W20260926 09:22:15.848318 27230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:38103 (error 108)
I20260926 09:22:15.848510 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27456
I20260926 09:22:15.849798 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:15.849543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38103 (local address 127.0.0.1:53564)
0926 09:22:15.849687 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:15.849744 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260926 09:22:15.849967 27230 catalog_manager.cc:4584] TS c5922fa6d5e24027a5910e32ae038c4d (127.24.235.65:38103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 922b6c5979334369a3d0e071b1ad65ca: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38103: connect: Connection refused (error 111)
I20260926 09:22:15.856045 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27596
I20260926 09:22:15.860605 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27212
2026-09-26T09:22:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5863 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:15 dist-test-slave-7zlw krb5kdc[27887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:15 dist-test-slave-7zlw krb5kdc[27887](info): set up 2 sockets
Sep 26 09:22:15 dist-test-slave-7zlw krb5kdc[27887](info): commencing operation
krb5kdc: starting...
W20260926 09:22:16.933007 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:17.064288 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45127
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:17.147964 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:17.148265 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:17.148312 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:17.149569 27900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:17.149645 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:17.149668 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:17.149690 27900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:17.149719 27900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:17.151263 27900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:17.151573 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:17.151815 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:17.157382 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.157356 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.157336 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:17.157335 27900 server_base.cc:1034] running on GCE node
I20260926 09:22:17.157838 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:17.158164 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:17.159340 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790414537159312 us; error 45 us; skew 500 ppm
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:17.162093 27900 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:17.162569 27900 webserver.cc:460] Webserver started at http://127.24.235.126:45667/ using document root <none> and password file <none>
I20260926 09:22:17.162762 27900 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:17.162804 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:17.162896 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:17.163909 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "373de33f16084c10a83cd03b14a0ed68"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "2fab2a87174fc61e8fac5263f368c8e0"
server_key_iv: "dcedeedd118f363b6d612140907faacf"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.164219 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "373de33f16084c10a83cd03b14a0ed68"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "2fab2a87174fc61e8fac5263f368c8e0"
server_key_iv: "dcedeedd118f363b6d612140907faacf"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.165695 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:17.166427 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.166620 27900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:17.166709 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "373de33f16084c10a83cd03b14a0ed68"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "2fab2a87174fc61e8fac5263f368c8e0"
server_key_iv: "dcedeedd118f363b6d612140907faacf"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.166764 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:17.176847 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:17.178857 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:17.178969 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:17.182906 27900 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:45127
I20260926 09:22:17.182964 27967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:45127 every 8 connection(s)
I20260926 09:22:17.183321 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:17.184345 27968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:17.186777 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68: Bootstrap starting.
I20260926 09:22:17.187378 27968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:17.187711 27968 log.cc:824] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:17.188508 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68: No bootstrap required, opened a new log
I20260926 09:22:17.189591 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27900
I20260926 09:22:17.189690 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:17.189867 25517 external_mini_cluster.cc:1428] Setting key 058100ad3d65ec34a5867849d942e2ca
I20260926 09:22:17.190814 27968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } }
I20260926 09:22:17.191001 27968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:17.191074 27968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373de33f16084c10a83cd03b14a0ed68, State: Initialized, Role: FOLLOWER
I20260926 09:22:17.191227 27968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } }
I20260926 09:22:17.191314 27968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:17.191344 27968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:17.191393 27968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:17.192373 27968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } }
I20260926 09:22:17.192471 27968 leader_election.cc:302] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 373de33f16084c10a83cd03b14a0ed68; no voters:
I20260926 09:22:17.192736 27968 leader_election.cc:288] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:17.192765 27973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:17.192901 27973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [term 1 LEADER]: Becoming Leader. State: Replica: 373de33f16084c10a83cd03b14a0ed68, State: Running, Role: LEADER
I20260926 09:22:17.193099 27973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } }
I20260926 09:22:17.193176 27968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:17.193563 27973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 373de33f16084c10a83cd03b14a0ed68. Latest consensus state: current_term: 1 leader_uuid: "373de33f16084c10a83cd03b14a0ed68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } } }
I20260926 09:22:17.193876 27973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:17.193787 27974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "373de33f16084c10a83cd03b14a0ed68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373de33f16084c10a83cd03b14a0ed68" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45127 } } }
I20260926 09:22:17.194351 27974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:17.194541 27979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:17.195144 27979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:17.196897 27979 catalog_manager.cc:1347] Generated new cluster ID: 3d3b48bc71e64867962fcab477e68442
I20260926 09:22:17.196959 27979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:17.201321 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.190447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.24.235.126:45127)
0926 09:22:17.190659 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:17.190667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:17.190693 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:22:17.190794 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:17.190800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:17.190843 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:17.190987 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:17.191792 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.192830 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.193934 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.194565 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.194696 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.197861 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:17.197865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:17.197866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:17.197879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:17.199590 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.199976 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.199980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.199980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.200093 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.200365 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.200366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.200366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.200653 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:17.200836 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:17.200934 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 09:22:17.201105 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260926 09:22:17.223428 27979 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:17.224368 27979 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:17.231693 27979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 373de33f16084c10a83cd03b14a0ed68: Generated new TSK 0
I20260926 09:22:17.231927 27979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:17.269284 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:17.356202 27994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:17.356434 27994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:17.356462 27994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:17.357791 27994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:17.357843 27994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:17.357872 27994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:17.359397 27994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:17.359681 27994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:17.359929 27994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:17.365358 28000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.365512 28002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.365394 27999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:17.366014 27994 server_base.cc:1034] running on GCE node
I20260926 09:22:17.366298 27994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:17.366631 27994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:17.367933 27994 hybrid_clock.cc:648] HybridClock initialized: now 1790414537367911 us; error 41 us; skew 500 ppm
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:17.371277 27994 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:17.371917 27994 webserver.cc:460] Webserver started at http://127.24.235.65:37415/ using document root <none> and password file <none>
I20260926 09:22:17.372285 27994 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:17.372359 27994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:17.372519 27994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:17.373731 27994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "88a9d88e64b240faa7eed11a319d2edb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "5297dafcb11acf4bc8ce5a9ccb6ba40e"
server_key_iv: "e0fccfdc5d34bd1a27413089bf6788d5"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.374112 27994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "88a9d88e64b240faa7eed11a319d2edb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "5297dafcb11acf4bc8ce5a9ccb6ba40e"
server_key_iv: "e0fccfdc5d34bd1a27413089bf6788d5"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.375559 27994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:17.376461 28009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.376688 27994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:17.376806 27994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "88a9d88e64b240faa7eed11a319d2edb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "5297dafcb11acf4bc8ce5a9ccb6ba40e"
server_key_iv: "e0fccfdc5d34bd1a27413089bf6788d5"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.376889 27994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:17.388196 27994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:17.390530 27994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:17.390714 27994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:17.390966 27994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:17.391299 27994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:17.391356 27994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.391400 27994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:17.391424 27994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.398013 27994 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:33635
I20260926 09:22:17.398052 28122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:33635 every 8 connection(s)
I20260926 09:22:17.398408 27994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:17.405160 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 27994
I20260926 09:22:17.405265 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:17.405467 25517 external_mini_cluster.cc:1428] Setting key 78bdf0d69b30e561e2e470b6e1418e24
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:17.409732 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.399242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:35707 (local address 127.24.235.126:45127)
0926 09:22:17.399315 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:17.399316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:17.399537 (+ 221us) server_negotiation.cc:407] Connection header received
0926 09:22:17.400360 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:17.400362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:17.400379 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:17.400452 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:17.401720 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.402256 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.403333 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.403546 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.403611 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.406274 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:17.406275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:17.406275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:17.406287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:17.407926 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.408523 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.408525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.408525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.408589 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.408896 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.408898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.408898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.409125 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:17.409209 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:17.409479 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 09:22:17.409585 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:22:17.409757 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.399215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45127 (local address 127.24.235.65:35707)
0926 09:22:17.399400 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:17.399445 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:17.400219 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:17.400586 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:17.400592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:17.401364 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:17.401571 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.401576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.402363 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.402364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.403232 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.403235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.403637 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.403638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.403746 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.404294 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:17.404304 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:17.406089 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:17.408024 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.408035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.408038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.408419 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.408669 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.408670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.408671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.408813 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.409198 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:17.409203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:17.409342 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 09:22:17.409434 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:22:17.409571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260926 09:22:17.410359 28123 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45127
I20260926 09:22:17.410481 28123 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:17.410686 28123 heartbeater.cc:511] Master 127.24.235.126:45127 requested a full tablet report, sending...
I20260926 09:22:17.411314 27932 ts_manager.cc:194] Registered new tserver with Master: 88a9d88e64b240faa7eed11a319d2edb (127.24.235.65:33635)
I20260926 09:22:17.412331 27932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:35707
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:17.436709 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:17.519940 28128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:17.520164 28128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:17.520190 28128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:17.521605 28128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:17.521656 28128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:17.521685 28128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:17.523240 28128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:17.523525 28128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:17.523788 28128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:17.529467 28133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.529580 28136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.529460 28134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:17.529795 28128 server_base.cc:1034] running on GCE node
I20260926 09:22:17.529980 28128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:17.530215 28128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:17.531370 28128 hybrid_clock.cc:648] HybridClock initialized: now 1790414537531358 us; error 25 us; skew 500 ppm
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:17.534209 28128 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:17.534627 28128 webserver.cc:460] Webserver started at http://127.24.235.66:42111/ using document root <none> and password file <none>
I20260926 09:22:17.534834 28128 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:17.534873 28128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:17.534977 28128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:17.536026 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "d5f351450afb4bbba72f30f75843f620"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "9a907b8ea44b8acc2f618bd906d98884"
server_key_iv: "d1b6098d6a56321196c69ddf21497dab"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.536329 28128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "d5f351450afb4bbba72f30f75843f620"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "9a907b8ea44b8acc2f618bd906d98884"
server_key_iv: "d1b6098d6a56321196c69ddf21497dab"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.537729 28128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:17.538618 28143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.538841 28128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:17.538950 28128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "d5f351450afb4bbba72f30f75843f620"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "9a907b8ea44b8acc2f618bd906d98884"
server_key_iv: "d1b6098d6a56321196c69ddf21497dab"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.539027 28128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:17.550624 28128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:17.553241 28128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:17.553438 28128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:17.553800 28128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:17.554124 28128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:17.554180 28128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.554224 28128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:17.554251 28128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.563264 28128 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:38555
I20260926 09:22:17.563344 28256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:38555 every 8 connection(s)
I20260926 09:22:17.563688 28128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:17.572328 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28128
I20260926 09:22:17.572417 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:17.572640 25517 external_mini_cluster.cc:1428] Setting key b0ba51a48e61a0e6054ba1f32cf3a2ae
I20260926 09:22:17.574273 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.564539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:56663 (local address 127.24.235.126:45127)
0926 09:22:17.564591 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:17.564593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:17.564762 (+ 169us) server_negotiation.cc:407] Connection header received
0926 09:22:17.565938 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:17.565939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:17.565949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:17.566012 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:17.566647 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.567152 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.568366 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.568620 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.568691 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.571266 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:17.571269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:17.571269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:17.571279 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:17.572638 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.573206 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.573208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.573208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.573271 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.573589 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.573590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.573590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.573712 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:17.573764 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:17.574138 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 09:22:17.574210 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:17.574613 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.564461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45127 (local address 127.24.235.66:56663)
0926 09:22:17.564627 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:17.564671 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:17.565841 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:17.566068 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:17.566071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:17.566353 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:17.566574 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.566578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.567264 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.567265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.568243 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.568246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.568739 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.568741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.568856 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.569560 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:17.569571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:17.571144 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:17.572753 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.572762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.572766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.573135 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.573346 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.573347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.573347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.573506 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.573780 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:17.573786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:17.573970 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 09:22:17.574105 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 09:22:17.574297 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260926 09:22:17.575603 28257 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45127
I20260926 09:22:17.575743 28257 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:17.575959 28257 heartbeater.cc:511] Master 127.24.235.126:45127 requested a full tablet report, sending...
I20260926 09:22:17.576388 27932 ts_manager.cc:194] Registered new tserver with Master: d5f351450afb4bbba72f30f75843f620 (127.24.235.66:38555)
I20260926 09:22:17.576917 27932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:56663
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:17.600066 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:17.689826 28262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:17.690044 28262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:17.690068 28262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:17.691267 28262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:17.691310 28262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:17.691336 28262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:17.692806 28262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:17.693084 28262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:17.693364 28262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:17.698740 28268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.698747 28267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:17.698747 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:17.699079 28262 server_base.cc:1034] running on GCE node
I20260926 09:22:17.699282 28262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:17.699570 28262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:17.700765 28262 hybrid_clock.cc:648] HybridClock initialized: now 1790414537700759 us; error 39 us; skew 500 ppm
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:17.703905 28262 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:17.704488 28262 webserver.cc:460] Webserver started at http://127.24.235.67:33197/ using document root <none> and password file <none>
I20260926 09:22:17.704769 28262 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:17.704860 28262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:17.705090 28262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:17.706383 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "c0bbd1123a344a97ba9de1c2161019bb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "8f8e67acba9c12b184108866285109c5"
server_key_iv: "ed65f2e76900969ca17c73aea9888f6b"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.706853 28262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "c0bbd1123a344a97ba9de1c2161019bb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "8f8e67acba9c12b184108866285109c5"
server_key_iv: "ed65f2e76900969ca17c73aea9888f6b"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.708518 28262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:17.709417 28277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.709697 28262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:17.709815 28262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "c0bbd1123a344a97ba9de1c2161019bb"
format_stamp: "Formatted at 2026-09-26 09:22:17 on dist-test-slave-7zlw"
server_key: "8f8e67acba9c12b184108866285109c5"
server_key_iv: "ed65f2e76900969ca17c73aea9888f6b"
server_key_version: "encryptionkey@0"
I20260926 09:22:17.709895 28262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:17.729622 28262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:17.732054 28262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:17.732223 28262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:17.732461 28262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:17.732780 28262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:17.732834 28262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.732879 28262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:17.732913 28262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:17.739085 28262 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:40469
I20260926 09:22:17.739168 28390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:40469 every 8 connection(s)
I20260926 09:22:17.739506 28262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[27887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414537, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:17.746058 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28262
I20260926 09:22:17.746150 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:17.746332 25517 external_mini_cluster.cc:1428] Setting key a5a44d8690b6389bae3aa24c027b23ef
I20260926 09:22:17.749495 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.740242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:60983 (local address 127.24.235.126:45127)
0926 09:22:17.740304 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:17.740306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:17.740575 (+ 269us) server_negotiation.cc:407] Connection header received
0926 09:22:17.741094 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:17.741095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:17.741111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:17.741181 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:17.742163 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.742671 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.743811 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.744016 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.744076 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.746394 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:17.746396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:17.746396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:17.746404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:17.747892 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.748294 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.748294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.748295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.748346 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:17.748613 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:17.748614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:17.748614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:17.748726 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:17.748780 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:17.749368 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 09:22:17.749432 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:22:17.749840 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:17.740200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45127 (local address 127.24.235.67:60983)
0926 09:22:17.740373 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:17.740414 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:17.740959 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:17.741239 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:17.741243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:17.741887 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:17.742086 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.742091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.742798 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.742799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.743669 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:17.743671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:17.744108 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:17.744109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:17.744223 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:17.744803 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:17.744810 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:17.746300 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:17.747985 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.747991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.747993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.748223 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.748417 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:17.748418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:17.748418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:17.748551 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:17.748979 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:17.748986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:17.749219 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 09:22:17.749346 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 09:22:17.749570 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 09:22:17.750409 28391 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45127
I20260926 09:22:17.750504 28391 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:17.750646 28391 heartbeater.cc:511] Master 127.24.235.126:45127 requested a full tablet report, sending...
I20260926 09:22:17.751003 27932 ts_manager.cc:194] Registered new tserver with Master: c0bbd1123a344a97ba9de1c2161019bb (127.24.235.67:40469)
I20260926 09:22:17.751400 27932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:60983
I20260926 09:22:17.758162 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:22:17.770251 27972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:17.769247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.24.235.126:45127)
0926 09:22:17.769362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:17.769364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:17.769378 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:22:17.769430 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:17.769431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:17.769444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:17.769513 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:17.770155 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54774: BlockingRecv error: recv got EOF from 127.0.0.1:54774 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 09:22:17.771056 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27994
I20260926 09:22:17.776455 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28128
I20260926 09:22:17.781703 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28262
I20260926 09:22:17.786281 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 27900
2026-09-26T09:22:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1923 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[28403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[28403](info): set up 2 sockets
Sep 26 09:22:17 dist-test-slave-7zlw krb5kdc[28403](info): commencing operation
krb5kdc: starting...
W20260926 09:22:18.853139 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:18 dist-test-slave-7zlw krb5kdc[28403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:18.981782 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:35187
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:19.068532 28416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:19.068758 28416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:19.068782 28416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:19.070006 28416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:19.070053 28416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:19.070068 28416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:19.070084 28416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:19.070096 28416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:19.071625 28416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:19.071899 28416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:19.072154 28416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:19.077273 28424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:19.077265 28422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:19.077344 28416 server_base.cc:1034] running on GCE node
W20260926 09:22:19.077265 28421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:19.077796 28416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:19.078094 28416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:19.079272 28416 hybrid_clock.cc:648] HybridClock initialized: now 1790414539079260 us; error 34 us; skew 500 ppm
Sep 26 09:22:19 dist-test-slave-7zlw krb5kdc[28403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414539, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:19.081964 28416 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:19.082458 28416 webserver.cc:460] Webserver started at http://127.24.235.126:33023/ using document root <none> and password file <none>
I20260926 09:22:19.082710 28416 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:19.082780 28416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:19.082928 28416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:19.084007 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "8e3f4f8701f948c58ada416b4c64a982"
format_stamp: "Formatted at 2026-09-26 09:22:19 on dist-test-slave-7zlw"
server_key: "b7e84b31d49540d2a5ed3c63bddaac05"
server_key_iv: "f726866799605e3c6b21b7b69818e207"
server_key_version: "encryptionkey@0"
I20260926 09:22:19.084376 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "8e3f4f8701f948c58ada416b4c64a982"
format_stamp: "Formatted at 2026-09-26 09:22:19 on dist-test-slave-7zlw"
server_key: "b7e84b31d49540d2a5ed3c63bddaac05"
server_key_iv: "f726866799605e3c6b21b7b69818e207"
server_key_version: "encryptionkey@0"
I20260926 09:22:19.085825 28416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:19.086645 28431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:19.086939 28416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:19.087047 28416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "8e3f4f8701f948c58ada416b4c64a982"
format_stamp: "Formatted at 2026-09-26 09:22:19 on dist-test-slave-7zlw"
server_key: "b7e84b31d49540d2a5ed3c63bddaac05"
server_key_iv: "f726866799605e3c6b21b7b69818e207"
server_key_version: "encryptionkey@0"
I20260926 09:22:19.087123 28416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:19.103956 28416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:19.106492 28416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:19.106675 28416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:19.111327 28416 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:35187
I20260926 09:22:19.111361 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:35187 every 8 connection(s)
I20260926 09:22:19.111949 28416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:19.112601 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:19.115219 28484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982: Bootstrap starting.
I20260926 09:22:19.115844 28484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:19.116173 28484 log.cc:824] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:19.116916 28484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982: No bootstrap required, opened a new log
I20260926 09:22:19.117530 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28416
I20260926 09:22:19.117615 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:19.117825 25517 external_mini_cluster.cc:1428] Setting key 9dc2611bfebf6af88fc7164997f0862f
I20260926 09:22:19.119587 28484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } }
I20260926 09:22:19.119756 28484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:19.119825 28484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3f4f8701f948c58ada416b4c64a982, State: Initialized, Role: FOLLOWER
I20260926 09:22:19.120007 28484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } }
I20260926 09:22:19.120105 28484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:19.120150 28484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:19.120199 28484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:19.120785 28484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } }
I20260926 09:22:19.120908 28484 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3f4f8701f948c58ada416b4c64a982; no voters:
I20260926 09:22:19.121104 28484 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:19.121150 28489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:19.121263 28489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [term 1 LEADER]: Becoming Leader. State: Replica: 8e3f4f8701f948c58ada416b4c64a982, State: Running, Role: LEADER
I20260926 09:22:19.121345 28489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } }
I20260926 09:22:19.121384 28484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:19.121708 28489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3f4f8701f948c58ada416b4c64a982. Latest consensus state: current_term: 1 leader_uuid: "8e3f4f8701f948c58ada416b4c64a982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } } }
I20260926 09:22:19.121733 28491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3f4f8701f948c58ada416b4c64a982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3f4f8701f948c58ada416b4c64a982" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 35187 } } }
I20260926 09:22:19.121877 28489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:19.121886 28491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:19.122138 28495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:19.122789 28495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:22:19 dist-test-slave-7zlw krb5kdc[28403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:19.130782 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:19.118402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.24.235.126:35187)
0926 09:22:19.118604 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:19.118612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:19.118671 (+ 59us) server_negotiation.cc:407] Connection header received
0926 09:22:19.119566 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:19.119572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:19.119624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:19.119811 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:19.120620 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:19.121995 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:19.123105 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:19.123713 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:19.123822 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:19.126281 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:19.126285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:19.126286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:19.126297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:19.128362 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:19.128753 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:19.128756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:19.128757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:19.129508 (+ 751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:19.129819 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:19.129820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:19.129821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:19.130162 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:19.130327 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:19.130402 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:22:19.130551 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260926 09:22:19.131500 28495 catalog_manager.cc:1347] Generated new cluster ID: 7dc398fe5c5242fdb9099115d5e00a72
I20260926 09:22:19.131642 28495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:19.131772 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:22:19.145227 28495 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:19.146029 28495 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:19.152139 28495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3f4f8701f948c58ada416b4c64a982: Generated new TSK 0
I20260926 09:22:19.152343 28495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:19.155041 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:19.151205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.24.235.126:35187)
0926 09:22:19.151251 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:19.151253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:19.151365 (+ 112us) server_negotiation.cc:407] Connection header received
0926 09:22:19.151423 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:19.151424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:19.151436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:19.151517 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:19.151986 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:19.152603 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:19.153447 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:19.153658 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:19.153776 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:19.154516 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:19.154517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:19.154518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:19.154765 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:19.154782 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:19.154789 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:19.154858 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:19.154923 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:22:19.154952 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:19.157691 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28416
2026-09-26T09:22:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1369 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:19 dist-test-slave-7zlw krb5kdc[28517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:19 dist-test-slave-7zlw krb5kdc[28517](info): set up 2 sockets
Sep 26 09:22:19 dist-test-slave-7zlw krb5kdc[28517](info): commencing operation
krb5kdc: starting...
W20260926 09:22:20.225303 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:20.353282 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:42885
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:20.437775 28530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:20.438072 28530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:20.438102 28530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:20.439344 28530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:20.439393 28530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:20.439407 28530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:20.439420 28530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:20.439432 28530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:20.441147 28530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:42885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:42885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:20.441424 28530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:20.441670 28530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:20.446907 28535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.446898 28538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.446898 28536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:20.447281 28530 server_base.cc:1034] running on GCE node
I20260926 09:22:20.447543 28530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:20.447875 28530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:20.449074 28530 hybrid_clock.cc:648] HybridClock initialized: now 1790414540449033 us; error 47 us; skew 500 ppm
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:20.451464 28530 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:20.451864 28530 webserver.cc:460] Webserver started at http://127.24.235.126:38959/ using document root <none> and password file <none>
I20260926 09:22:20.452059 28530 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:20.452101 28530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:20.452188 28530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:20.453301 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "527f993138c843c38993bac67c05f8cf"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "492b8030b064bbb4d2256ea0b1b69468"
server_key_iv: "65bed3c2bd267ac2cf60bd4660de5d7d"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.453608 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "527f993138c843c38993bac67c05f8cf"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "492b8030b064bbb4d2256ea0b1b69468"
server_key_iv: "65bed3c2bd267ac2cf60bd4660de5d7d"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.454962 28530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:20.455773 28545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.456018 28530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:20.456116 28530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "527f993138c843c38993bac67c05f8cf"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "492b8030b064bbb4d2256ea0b1b69468"
server_key_iv: "65bed3c2bd267ac2cf60bd4660de5d7d"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.456203 28530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:20.472564 28530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:20.475546 28530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:20.475769 28530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:20.482276 28530 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:42885
I20260926 09:22:20.482347 28597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:42885 every 8 connection(s)
I20260926 09:22:20.482802 28530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:20.484321 28598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:20.487636 28598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf: Bootstrap starting.
I20260926 09:22:20.488210 28598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:20.488456 28598 log.cc:824] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:20.488797 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28530
I20260926 09:22:20.488963 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:20.489181 25517 external_mini_cluster.cc:1428] Setting key 6301aa1a9a4e919ef80f448a9b9cbe42
I20260926 09:22:20.489327 28598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf: No bootstrap required, opened a new log
I20260926 09:22:20.491307 28598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } }
I20260926 09:22:20.491446 28598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:20.491467 28598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527f993138c843c38993bac67c05f8cf, State: Initialized, Role: FOLLOWER
I20260926 09:22:20.491614 28598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } }
I20260926 09:22:20.491725 28598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:20.491751 28598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:20.491816 28598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:20.492406 28598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } }
I20260926 09:22:20.492488 28598 leader_election.cc:302] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527f993138c843c38993bac67c05f8cf; no voters:
I20260926 09:22:20.492676 28598 leader_election.cc:288] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:20.492712 28603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:20.492833 28603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [term 1 LEADER]: Becoming Leader. State: Replica: 527f993138c843c38993bac67c05f8cf, State: Running, Role: LEADER
I20260926 09:22:20.492916 28603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } }
I20260926 09:22:20.492969 28598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:20.493326 28603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527f993138c843c38993bac67c05f8cf. Latest consensus state: current_term: 1 leader_uuid: "527f993138c843c38993bac67c05f8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } } }
I20260926 09:22:20.493398 28603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:20.493646 28608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:20.495066 28604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527f993138c843c38993bac67c05f8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527f993138c843c38993bac67c05f8cf" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42885 } } }
I20260926 09:22:20.495091 28608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:20.495158 28604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:20.497150 28608 catalog_manager.cc:1347] Generated new cluster ID: d6eed153484048f1a70e42ac16fb95a0
I20260926 09:22:20.497218 28608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:20.502558 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:20.489767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.24.235.126:42885)
0926 09:22:20.489969 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:20.489977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:20.490004 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:22:20.490655 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:20.490663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:20.490717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:20.490881 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:20.491566 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.493082 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.494162 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.494966 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.495068 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:20.497975 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:20.497979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:20.497980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:20.497993 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:20.500057 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.500526 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.500529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.500529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.500637 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.500847 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.500848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.500848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.501308 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:20.501479 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:20.502170 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 09:22:20.502328 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260926 09:22:20.503896 28608 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:20.504684 28608 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:20.511937 28608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527f993138c843c38993bac67c05f8cf: Generated new TSK 0
I20260926 09:22:20.512193 28608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:20.548779 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:20.656255 28624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:20.656467 28624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:20.656491 28624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:20.657735 28624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:20.657779 28624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:20.657806 28624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:20.659366 28624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:20.659611 28624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:20.659830 28624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:20.665963 28632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.665973 28629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:20.666203 28624 server_base.cc:1034] running on GCE node
W20260926 09:22:20.665973 28630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:20.666550 28624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:20.666821 28624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:20.667987 28624 hybrid_clock.cc:648] HybridClock initialized: now 1790414540667977 us; error 37 us; skew 500 ppm
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:20.670636 28624 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:20.671082 28624 webserver.cc:460] Webserver started at http://127.24.235.65:44885/ using document root <none> and password file <none>
I20260926 09:22:20.671334 28624 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:20.671401 28624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:20.671541 28624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:20.672601 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "6d4330aa34ff4fc881e27b0201f9fc61"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "0c834ab1cefd7bb5a6ff0321b4329fa2"
server_key_iv: "109b277dd85d0aca6703fff0bc43c83f"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.672971 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "6d4330aa34ff4fc881e27b0201f9fc61"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "0c834ab1cefd7bb5a6ff0321b4329fa2"
server_key_iv: "109b277dd85d0aca6703fff0bc43c83f"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.674396 28624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:20.675216 28639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.675405 28624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:20.675498 28624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "6d4330aa34ff4fc881e27b0201f9fc61"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "0c834ab1cefd7bb5a6ff0321b4329fa2"
server_key_iv: "109b277dd85d0aca6703fff0bc43c83f"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.675580 28624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:20.683482 28624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:20.685761 28624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:20.685915 28624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:20.686148 28624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:20.686460 28624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:20.686513 28624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.686555 28624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:20.686581 28624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.692392 28624 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:43249
I20260926 09:22:20.692445 28752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:43249 every 8 connection(s)
I20260926 09:22:20.692792 28624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:20.695036 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28624
I20260926 09:22:20.695124 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:20.695344 25517 external_mini_cluster.cc:1428] Setting key 26a9609be4d7519f8cd5290b9e18b588
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:20.704020 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:20.693676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:48219 (local address 127.24.235.126:42885)
0926 09:22:20.693725 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:20.693728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:20.694049 (+ 321us) server_negotiation.cc:407] Connection header received
0926 09:22:20.694620 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:20.694621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:20.694634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:20.694713 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:20.695844 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.696534 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.697798 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.698133 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.698216 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:20.700725 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:20.700727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:20.700728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:20.700738 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:20.702550 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.702963 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.702965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.702966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.703025 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.703291 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.703292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.703293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.703448 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:20.703537 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:20.703808 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 09:22:20.703908 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:20.704054 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:20.693685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42885 (local address 127.24.235.65:48219)
0926 09:22:20.693896 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:20.693939 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:20.694472 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:20.694772 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:20.694776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:20.695469 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:20.695755 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:20.695760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.696639 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.696640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:20.697702 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:20.697705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.698227 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.698228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:20.698334 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:20.699033 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:20.699042 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:20.700622 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:20.702639 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:20.702646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:20.702648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:20.702883 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:20.703091 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:20.703092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:20.703092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:20.703216 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:20.703525 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:20.703530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:20.703665 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 09:22:20.703752 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:22:20.703881 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:22:20.704658 28753 heartbeater.cc:347] Connected to a master server at 127.24.235.126:42885
I20260926 09:22:20.704759 28753 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:20.704988 28753 heartbeater.cc:511] Master 127.24.235.126:42885 requested a full tablet report, sending...
I20260926 09:22:20.705643 28562 ts_manager.cc:194] Registered new tserver with Master: 6d4330aa34ff4fc881e27b0201f9fc61 (127.24.235.65:43249)
I20260926 09:22:20.706275 28562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:48219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:20.724531 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:20.808074 28758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:20.808315 28758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:20.808341 28758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:20.809646 28758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:20.809691 28758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:20.809719 28758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:20.811211 28758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:20.811467 28758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:20.811715 28758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:20.817095 28766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.817098 28763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.817090 28764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:20.817343 28758 server_base.cc:1034] running on GCE node
I20260926 09:22:20.817636 28758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:20.817910 28758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:20.819056 28758 hybrid_clock.cc:648] HybridClock initialized: now 1790414540819024 us; error 50 us; skew 500 ppm
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:20.821614 28758 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:20.822036 28758 webserver.cc:460] Webserver started at http://127.24.235.66:35243/ using document root <none> and password file <none>
I20260926 09:22:20.822229 28758 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:20.822271 28758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:20.822364 28758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:20.823403 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "ed18f99750164335b65f60cac923f495"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "106e9d767bcf0e207388778bb21762a2"
server_key_iv: "5933d5c89fa03cc5ebee33d66b485410"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.823719 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "ed18f99750164335b65f60cac923f495"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "106e9d767bcf0e207388778bb21762a2"
server_key_iv: "5933d5c89fa03cc5ebee33d66b485410"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.825155 28758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:20.825959 28773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.826182 28758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:20.826273 28758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "ed18f99750164335b65f60cac923f495"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "106e9d767bcf0e207388778bb21762a2"
server_key_iv: "5933d5c89fa03cc5ebee33d66b485410"
server_key_version: "encryptionkey@0"
I20260926 09:22:20.826333 28758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:20.834226 28758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:20.836225 28758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:20.836344 28758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:20.836545 28758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:20.836793 28758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:20.836822 28758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.836844 28758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:20.836856 28758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:20.842965 28758 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:45129
I20260926 09:22:20.843026 28886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:45129 every 8 connection(s)
I20260926 09:22:20.843375 28758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:20.850046 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28758
I20260926 09:22:20.850193 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:20.850401 25517 external_mini_cluster.cc:1428] Setting key 3a44b75c51e5240a59a25da1983d4888
I20260926 09:22:20.854347 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:20.844218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:59695 (local address 127.24.235.126:42885)
0926 09:22:20.844319 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:20.844322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:20.844567 (+ 245us) server_negotiation.cc:407] Connection header received
0926 09:22:20.845215 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:20.845216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:20.845231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:20.845304 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:20.846318 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.846852 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.847942 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.848143 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.848258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:20.850816 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:20.850818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:20.850818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:20.850828 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:20.852692 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.853239 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.853240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.853240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.853301 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:20.853637 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:20.853638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:20.853639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:20.853760 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:20.853819 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:20.854202 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 09:22:20.854270 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 09:22:20.854686 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:20.844098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42885 (local address 127.24.235.66:59695)
0926 09:22:20.844360 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:20.844408 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:20.845007 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:20.845380 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:20.845386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:20.845990 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:20.846192 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:20.846198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.846979 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.846981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:20.847828 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:20.847830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:20.848268 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:20.848269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:20.848401 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:20.848992 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:20.849000 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:20.850693 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:20.852787 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:20.852794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:20.852796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:20.853153 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:20.853373 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:20.853374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:20.853374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:20.853521 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:20.853829 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:20.853836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:20.854021 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 09:22:20.854135 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:22:20.854324 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":117,"threads_started":1}
I20260926 09:22:20.855223 28887 heartbeater.cc:347] Connected to a master server at 127.24.235.126:42885
I20260926 09:22:20.855345 28887 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:20.855553 28887 heartbeater.cc:511] Master 127.24.235.126:42885 requested a full tablet report, sending...
I20260926 09:22:20.855948 28562 ts_manager.cc:194] Registered new tserver with Master: ed18f99750164335b65f60cac923f495 (127.24.235.66:45129)
I20260926 09:22:20.856338 28562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:59695
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:20.881294 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:20.978487 28892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:20.978732 28892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:20.978757 28892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:20.979995 28892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:20.980046 28892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:20.980096 28892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:20.981703 28892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:42885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:20.981966 28892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:20.982229 28892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:20.987888 28898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.988073 28897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:20.988094 28900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:20.988250 28892 server_base.cc:1034] running on GCE node
I20260926 09:22:20.988457 28892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:20.988734 28892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:20.989923 28892 hybrid_clock.cc:648] HybridClock initialized: now 1790414540989910 us; error 38 us; skew 500 ppm
Sep 26 09:22:20 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:20.999745 28892 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:21.000430 28892 webserver.cc:460] Webserver started at http://127.24.235.67:34435/ using document root <none> and password file <none>
I20260926 09:22:21.000778 28892 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:21.000856 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:21.000994 28892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:21.002557 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "c9a0b1efeeac4d8e895fd9c01225420f"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "6ccffe63f168bf3af4aa1cb05f0d0e15"
server_key_iv: "3340dcdca8b059c07fcdf9e3197cae42"
server_key_version: "encryptionkey@0"
I20260926 09:22:21.003013 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a0b1efeeac4d8e895fd9c01225420f"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "6ccffe63f168bf3af4aa1cb05f0d0e15"
server_key_iv: "3340dcdca8b059c07fcdf9e3197cae42"
server_key_version: "encryptionkey@0"
I20260926 09:22:21.004626 28892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:21.005681 28907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:21.005905 28892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:21.006001 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "c9a0b1efeeac4d8e895fd9c01225420f"
format_stamp: "Formatted at 2026-09-26 09:22:20 on dist-test-slave-7zlw"
server_key: "6ccffe63f168bf3af4aa1cb05f0d0e15"
server_key_iv: "3340dcdca8b059c07fcdf9e3197cae42"
server_key_version: "encryptionkey@0"
I20260926 09:22:21.006083 28892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:21.025996 28892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:21.028088 28892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:21.028226 28892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:21.028438 28892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:21.028780 28892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:21.028816 28892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:21.028841 28892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:21.028896 28892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:21.035475 28892 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:44317
I20260926 09:22:21.035574 29020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:44317 every 8 connection(s)
I20260926 09:22:21.035933 28892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:21.039115 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 28892
I20260926 09:22:21.039223 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:21.039395 25517 external_mini_cluster.cc:1428] Setting key 46e5d449db429510de80369a7527243f
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414540, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:21.046515 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:21.036860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:41489 (local address 127.24.235.126:42885)
0926 09:22:21.036928 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:21.036930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:21.037182 (+ 252us) server_negotiation.cc:407] Connection header received
0926 09:22:21.037752 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:21.037754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:21.037767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:21.037841 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:21.038861 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.039653 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.040948 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.041265 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.041334 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:21.043706 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:21.043707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:21.043708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:21.043715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:21.045258 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:21.045667 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:21.045668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:21.045669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:21.045719 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:21.045980 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:21.045981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:21.045981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:21.046080 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:21.046130 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:21.046386 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 09:22:21.046448 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:21.046837 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:21.036800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42885 (local address 127.24.235.67:41489)
0926 09:22:21.037000 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:21.037074 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:21.037617 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:21.037957 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:21.037961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:21.038537 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:21.038736 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:21.038741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.039755 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.039757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:21.040843 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:21.040846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.041354 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.041355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:21.041468 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:21.042069 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:21.042077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:21.043611 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:21.045329 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:21.045335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:21.045337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:21.045586 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:21.045802 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:21.045802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:21.045803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:21.045909 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:21.046139 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:21.046144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:21.046264 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 09:22:21.046351 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:22:21.046472 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 09:22:21.047365 29021 heartbeater.cc:347] Connected to a master server at 127.24.235.126:42885
I20260926 09:22:21.047462 29021 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:21.047674 29021 heartbeater.cc:511] Master 127.24.235.126:42885 requested a full tablet report, sending...
I20260926 09:22:21.048050 28562 ts_manager.cc:194] Registered new tserver with Master: c9a0b1efeeac4d8e895fd9c01225420f (127.24.235.67:44317)
I20260926 09:22:21.048430 28562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:41489
I20260926 09:22:21.051618 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414541, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414541, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:21.075299 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:21.067847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.24.235.126:42885)
0926 09:22:21.067914 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:21.067917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:21.068050 (+ 133us) server_negotiation.cc:407] Connection header received
0926 09:22:21.068113 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:21.068114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:21.068124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:21.068214 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:21.068856 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.069523 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.070287 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:21.070464 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:21.070520 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:21.072884 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:21.072885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:21.072886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:21.072892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:21.074345 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:21.074666 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:21.074667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:21.074667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:21.074716 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:21.074873 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:21.074874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:21.074874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:21.075012 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:21.075087 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:21.075170 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 09:22:21.075229 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 09:22:21.075562 28562 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52820
I20260926 09:22:21.076242 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28624
I20260926 09:22:21.080590 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28758
I20260926 09:22:21.085431 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28892
I20260926 09:22:21.089975 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 28530
2026-09-26T09:22:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1934 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[29033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[29033](info): set up 2 sockets
Sep 26 09:22:21 dist-test-slave-7zlw krb5kdc[29033](info): commencing operation
krb5kdc: starting...
W20260926 09:22:22.141194 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:22.272634 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41627
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:22.355985 29046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:22.356217 29046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:22.356240 29046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:22.357470 29046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:22.357520 29046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:22.357535 29046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:22.357548 29046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:22.357560 29046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:22.359550 29046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:22.359851 29046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:22.360148 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:22.365911 29051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.365958 29046 server_base.cc:1034] running on GCE node
W20260926 09:22:22.366066 29054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:22.366045 29052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.366338 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:22.366677 29046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:22.367874 29046 hybrid_clock.cc:648] HybridClock initialized: now 1790414542367853 us; error 42 us; skew 500 ppm
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:22.370782 29046 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:22.371317 29046 webserver.cc:460] Webserver started at http://127.24.235.126:37203/ using document root <none> and password file <none>
I20260926 09:22:22.371567 29046 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:22.371690 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:22.371850 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:22.372965 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "c9502b12c69504ebcf97e00571b751a0"
server_key_iv: "38a8bf457786af8d790e122da98cba1a"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.373427 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "c9502b12c69504ebcf97e00571b751a0"
server_key_iv: "38a8bf457786af8d790e122da98cba1a"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.374882 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:22.375736 29061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.375952 29046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:22.376057 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "c9502b12c69504ebcf97e00571b751a0"
server_key_iv: "38a8bf457786af8d790e122da98cba1a"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.376157 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:22.416277 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:22.418845 29046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:22.419028 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:22.423188 29046 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41627
I20260926 09:22:22.423243 29113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41627 every 8 connection(s)
I20260926 09:22:22.423803 29046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:22.424482 29114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:22.427006 29114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d: Bootstrap starting.
I20260926 09:22:22.427577 29114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:22.427829 29114 log.cc:824] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:22.428423 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29046
I20260926 09:22:22.428507 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:22.428640 25517 external_mini_cluster.cc:1428] Setting key e37a0138ecbf2ec1e5bdca2f5b9d7b8a
I20260926 09:22:22.428825 29114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d: No bootstrap required, opened a new log
I20260926 09:22:22.431735 29114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } }
I20260926 09:22:22.431908 29114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:22.431941 29114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2c041ac0bf43a7b6ad9a2adc14ab1d, State: Initialized, Role: FOLLOWER
I20260926 09:22:22.432055 29114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } }
I20260926 09:22:22.432126 29114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:22.432155 29114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:22.432191 29114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:22.432868 29114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } }
I20260926 09:22:22.432955 29114 leader_election.cc:302] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b2c041ac0bf43a7b6ad9a2adc14ab1d; no voters:
I20260926 09:22:22.433182 29114 leader_election.cc:288] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:22.433238 29119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:22.433372 29119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [term 1 LEADER]: Becoming Leader. State: Replica: 1b2c041ac0bf43a7b6ad9a2adc14ab1d, State: Running, Role: LEADER
I20260926 09:22:22.433456 29114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:22.433465 29119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } }
I20260926 09:22:22.433851 29119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b2c041ac0bf43a7b6ad9a2adc14ab1d. Latest consensus state: current_term: 1 leader_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } } }
I20260926 09:22:22.434120 29119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:22.434062 29120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2c041ac0bf43a7b6ad9a2adc14ab1d" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41627 } } }
I20260926 09:22:22.434274 29120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:22.434916 29123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:22.435446 29123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:22.437122 29123 catalog_manager.cc:1347] Generated new cluster ID: 0950243a573d4a9a8dd83d9245cf262c
I20260926 09:22:22.437177 29123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:22.440864 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.429230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.24.235.126:41627)
0926 09:22:22.429422 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.429432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.429468 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:22:22.430330 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.430339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.430403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.430557 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.431222 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.432593 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.433646 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.434259 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.434344 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.437446 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.437450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.437450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.437468 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.439160 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.439538 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.439541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.439544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.439628 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.439847 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.439848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.439848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.440075 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.440253 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.440331 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.440479 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260926 09:22:22.458794 29123 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:22.459432 29123 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:22.465646 29123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b2c041ac0bf43a7b6ad9a2adc14ab1d: Generated new TSK 0
I20260926 09:22:22.465859 29123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:22.502089 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:22.587675 29140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:22.587893 29140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:22.587917 29140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:22.589129 29140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:22.589174 29140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:22.589201 29140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:22.590745 29140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:22.591001 29140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:22.591271 29140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:22.596498 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:22.596515 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:22.596591 29148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.596808 29140 server_base.cc:1034] running on GCE node
I20260926 09:22:22.597131 29140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:22.597411 29140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:22.598578 29140 hybrid_clock.cc:648] HybridClock initialized: now 1790414542598552 us; error 40 us; skew 500 ppm
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:22.601294 29140 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:22.601752 29140 webserver.cc:460] Webserver started at http://127.24.235.65:42651/ using document root <none> and password file <none>
I20260926 09:22:22.601982 29140 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:22.602061 29140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:22.602214 29140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:22.603315 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "f836222033294874a2553e230734ba91"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "1070e334e47edfb75a9f68c6fd3e12b5"
server_key_iv: "c0e17c09bf7ad0887540ba1da12eb84b"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.603703 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "f836222033294874a2553e230734ba91"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "1070e334e47edfb75a9f68c6fd3e12b5"
server_key_iv: "c0e17c09bf7ad0887540ba1da12eb84b"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.605228 29140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:22.606034 29155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.606208 29140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:22.606308 29140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "f836222033294874a2553e230734ba91"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "1070e334e47edfb75a9f68c6fd3e12b5"
server_key_iv: "c0e17c09bf7ad0887540ba1da12eb84b"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.606391 29140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:22.617687 29140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:22.620007 29140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:22.620203 29140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:22.620460 29140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:22.620798 29140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:22.620855 29140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.620903 29140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:22.620931 29140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.627004 29140 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:45487
I20260926 09:22:22.627048 29268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:45487 every 8 connection(s)
I20260926 09:22:22.627398 29140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:22.628064 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29140
I20260926 09:22:22.628198 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:22.628394 25517 external_mini_cluster.cc:1428] Setting key 3a5ac91ece54f59d70b542ecd714389f
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:22:22.638875 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.628238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:60185 (local address 127.24.235.126:41627)
0926 09:22:22.628293 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.628296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.628597 (+ 301us) server_negotiation.cc:407] Connection header received
0926 09:22:22.629836 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.629837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.629848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.629903 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.630846 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.631647 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.632722 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.633096 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.633176 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.635803 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.635804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.635805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.635814 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.637499 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.637952 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.637953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.637953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.638015 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.638270 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.638271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.638272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.638384 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.638441 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.638734 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.638805 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:22:22.639094 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.628215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41627 (local address 127.24.235.65:60185)
0926 09:22:22.628393 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:22.628486 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:22.629711 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:22.629983 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:22.629987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:22.630434 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:22.630721 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.630728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.631803 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.631804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.632606 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.632608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.633180 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.633181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.633295 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.633879 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:22.633887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:22.635653 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:22.637590 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.637596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.637598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.637837 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.638092 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.638093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.638093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.638197 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.638449 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:22.638453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:22.638572 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:22:22.638686 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:22:22.638816 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260926 09:22:22.639605 29269 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41627
I20260926 09:22:22.639700 29269 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:22.639887 29269 heartbeater.cc:511] Master 127.24.235.126:41627 requested a full tablet report, sending...
I20260926 09:22:22.640530 29078 ts_manager.cc:194] Registered new tserver with Master: f836222033294874a2553e230734ba91 (127.24.235.65:45487)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
I20260926 09:22:22.641418 29078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:60185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:22.656423 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:22.738535 29274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:22.738761 29274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:22.738787 29274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:22.739995 29274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:22.740042 29274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:22.740069 29274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:22.741683 29274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:22.741953 29274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:22.742197 29274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:22.747313 29279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.747450 29274 server_base.cc:1034] running on GCE node
W20260926 09:22:22.747306 29282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:22.747340 29280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.747753 29274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:22.748005 29274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:22.749177 29274 hybrid_clock.cc:648] HybridClock initialized: now 1790414542749174 us; error 62 us; skew 500 ppm
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:22.751984 29274 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:22.752506 29274 webserver.cc:460] Webserver started at http://127.24.235.66:33635/ using document root <none> and password file <none>
I20260926 09:22:22.752763 29274 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:22.752836 29274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:22.752995 29274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:22.754091 29274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "0c49d68d4f174f2e8d7d4826db51ece5"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "8192936e20ef24297a2d92572ab20e91"
server_key_iv: "2eb1dc06009bd0f0826947c61ad86525"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.754456 29274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "0c49d68d4f174f2e8d7d4826db51ece5"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "8192936e20ef24297a2d92572ab20e91"
server_key_iv: "2eb1dc06009bd0f0826947c61ad86525"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.755810 29274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:22.756588 29289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.756784 29274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:22.756884 29274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "0c49d68d4f174f2e8d7d4826db51ece5"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "8192936e20ef24297a2d92572ab20e91"
server_key_iv: "2eb1dc06009bd0f0826947c61ad86525"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.756968 29274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:22.766960 29274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:22.768936 29274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:22.769114 29274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:22.769341 29274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:22.769665 29274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:22.769717 29274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.769753 29274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:22.769781 29274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.775859 29274 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:44373
I20260926 09:22:22.775905 29402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:44373 every 8 connection(s)
I20260926 09:22:22.776230 29274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:22.781833 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29274
I20260926 09:22:22.781950 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:22.782184 25517 external_mini_cluster.cc:1428] Setting key abb8b9440ac50e035007b87d009824bb
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:22.786317 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.777003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:36679 (local address 127.24.235.126:41627)
0926 09:22:22.777090 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.777094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.777389 (+ 295us) server_negotiation.cc:407] Connection header received
0926 09:22:22.778294 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.778296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.778306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.778398 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.779043 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.779565 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.780624 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.780813 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.780886 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.783381 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.783383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.783383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.783391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.784858 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.785315 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.785317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.785317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.785381 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.785655 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.785656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.785656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.785770 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.785846 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.786134 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.786212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:22:22.786533 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.776966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41627 (local address 127.24.235.66:36679)
0926 09:22:22.777275 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:22.777307 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:22.778193 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:22.778479 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:22.778483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:22.778742 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:22.778931 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.778934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.779685 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.779686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.780507 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.780509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.780897 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.780897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.781001 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.781531 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:22.781540 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:22.783280 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:22.784963 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.784977 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.784980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.785236 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.785463 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.785463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.785464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.785580 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.785838 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:22.785844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:22.785997 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 09:22:22.786099 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:22:22.786254 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":164,"threads_started":1}
I20260926 09:22:22.787101 29403 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41627
I20260926 09:22:22.787181 29403 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:22.787321 29403 heartbeater.cc:511] Master 127.24.235.126:41627 requested a full tablet report, sending...
I20260926 09:22:22.787678 29078 ts_manager.cc:194] Registered new tserver with Master: 0c49d68d4f174f2e8d7d4826db51ece5 (127.24.235.66:44373)
I20260926 09:22:22.788004 29078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:36679
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:22.810063 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:22.892231 29408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:22.892442 29408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:22.892467 29408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:22.893597 29408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:22.893633 29408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:22.893666 29408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:22.895118 29408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:22.895313 29408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:22.895498 29408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:22.900698 29416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:22.900708 29414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.900941 29408 server_base.cc:1034] running on GCE node
W20260926 09:22:22.900710 29413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:22.901309 29408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:22.901618 29408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:22.902894 29408 hybrid_clock.cc:648] HybridClock initialized: now 1790414542902872 us; error 36 us; skew 500 ppm
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:22.905833 29408 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:22.906353 29408 webserver.cc:460] Webserver started at http://127.24.235.67:34211/ using document root <none> and password file <none>
I20260926 09:22:22.906603 29408 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:22.906672 29408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:22.906831 29408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:22.908022 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "a5b3c68bc5b64cb2a4f9602f18245139"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "f4e00629130135b0c1fc6183d5a0e246"
server_key_iv: "abeeee00517a721cd24d2d8dc4076d42"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.908416 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b3c68bc5b64cb2a4f9602f18245139"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "f4e00629130135b0c1fc6183d5a0e246"
server_key_iv: "abeeee00517a721cd24d2d8dc4076d42"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.910040 29408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:22.911048 29423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.911242 29408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:22.911353 29408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "a5b3c68bc5b64cb2a4f9602f18245139"
format_stamp: "Formatted at 2026-09-26 09:22:22 on dist-test-slave-7zlw"
server_key: "f4e00629130135b0c1fc6183d5a0e246"
server_key_iv: "abeeee00517a721cd24d2d8dc4076d42"
server_key_version: "encryptionkey@0"
I20260926 09:22:22.911432 29408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:22.926960 29408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:22.929682 29408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:22.929857 29408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:22.930121 29408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:22.930457 29408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:22.930512 29408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.930554 29408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:22.930580 29408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:22.937096 29408 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:34581
I20260926 09:22:22.937148 29536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:34581 every 8 connection(s)
I20260926 09:22:22.937502 29408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:22.945575 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29408
I20260926 09:22:22.945690 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:22.945907 25517 external_mini_cluster.cc:1428] Setting key deca2c03392b1f9aebd64ba9ff8ac86c
I20260926 09:22:22.947875 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.938307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:33605 (local address 127.24.235.126:41627)
0926 09:22:22.938374 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.938375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.938593 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:22:22.939595 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.939596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.939610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.939676 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.940327 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.940838 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.941943 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.942156 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.942242 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.944418 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.944420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.944420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.944428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.946240 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.946839 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.946840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.946841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.946911 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.947186 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.947187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.947187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.947321 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.947396 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.947697 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.947783 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:22.948094 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.938295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41627 (local address 127.24.235.67:33605)
0926 09:22:22.938472 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:22.938509 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:22.939471 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:22.939751 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:22.939755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:22.940028 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:22.940230 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.940234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.940939 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.940940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.941837 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:22.941840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.942250 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.942251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:22.942385 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.942895 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:22.942902 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:22.944310 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:22.946398 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.946405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.946408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.946740 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.946980 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:22.946981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:22.946981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:22.947098 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:22.947397 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:22.947405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:22.947551 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 09:22:22.947662 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 09:22:22.947826 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260926 09:22:22.948724 29537 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41627
I20260926 09:22:22.948825 29537 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:22.948978 29537 heartbeater.cc:511] Master 127.24.235.126:41627 requested a full tablet report, sending...
I20260926 09:22:22.949538 29078 ts_manager.cc:194] Registered new tserver with Master: a5b3c68bc5b64cb2a4f9602f18245139 (127.24.235.67:34581)
I20260926 09:22:22.949946 29078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:33605
I20260926 09:22:22.957856 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:22.982728 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.974382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.24.235.65:45487)
0926 09:22:22.974603 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.974608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.974619 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:22.974647 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.974649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.974662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.974739 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.975327 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.976102 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.976853 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.977282 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.977352 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.979883 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.979891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.979891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.979898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.981476 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.981871 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.981874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.981874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.981930 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.982113 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.982113 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.982114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.982245 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.982349 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.982568 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.982641 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
Sep 26 09:22:22 dist-test-slave-7zlw krb5kdc[29033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:22.991462 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:22.984345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.24.235.126:41627)
0926 09:22:22.984415 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:22.984417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:22.984429 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:22.984513 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:22.984514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:22.984524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:22.984609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:22.985190 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.985704 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.986399 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:22.986601 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:22.986689 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:22.988934 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:22.988935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:22.988935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:22.988941 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:22.990399 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.990734 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.990735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.990735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.990784 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:22.990979 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:22.990980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:22.990980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:22.991148 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:22.991253 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:22.991322 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:22:22.991397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 09:22:22.991832 29078 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48914
I20260926 09:22:22.992254 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29140
I20260926 09:22:22.997017 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29274
I20260926 09:22:23.005395 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29408
I20260926 09:22:23.014608 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29046
2026-09-26T09:22:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1926 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:23 dist-test-slave-7zlw krb5kdc[29549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:23 dist-test-slave-7zlw krb5kdc[29549](info): set up 2 sockets
Sep 26 09:22:23 dist-test-slave-7zlw krb5kdc[29549](info): commencing operation
krb5kdc: starting...
W20260926 09:22:24.057224 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:24.188241 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33849
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:24.270835 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:24.271086 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:24.271121 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:24.272271 29562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:24.272346 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:24.272370 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:24.272392 29562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:24.272421 29562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:24.273983 29562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:24.274264 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:24.274500 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:24.279918 29567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.279934 29568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.280073 29562 server_base.cc:1034] running on GCE node
W20260926 09:22:24.279918 29570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.280476 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:24.280740 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:24.281903 29562 hybrid_clock.cc:648] HybridClock initialized: now 1790414544281886 us; error 44 us; skew 500 ppm
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:24.284403 29562 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:24.284816 29562 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:24.284891 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:24.285079 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:24.286129 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "fa7ed5905f194d2683c222f60d056a61"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "af8f474cf879cda85cc0eae44fd3036e"
server_key_iv: "b2af01b98fd4dc8206edaf58d3baffab"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.286506 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "fa7ed5905f194d2683c222f60d056a61"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "af8f474cf879cda85cc0eae44fd3036e"
server_key_iv: "b2af01b98fd4dc8206edaf58d3baffab"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.287850 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:24.288806 29576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.289088 29562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:24.289212 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "fa7ed5905f194d2683c222f60d056a61"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "af8f474cf879cda85cc0eae44fd3036e"
server_key_iv: "b2af01b98fd4dc8206edaf58d3baffab"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.289302 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:24.302042 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:24.304181 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:24.304351 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:24.308566 29562 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33849
I20260926 09:22:24.308602 29628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33849 every 8 connection(s)
I20260926 09:22:24.308954 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:24.310014 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:24.312444 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: Bootstrap starting.
I20260926 09:22:24.313037 29629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:24.313366 29629 log.cc:824] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:24.313515 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29562
I20260926 09:22:24.313634 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:24.313867 25517 external_mini_cluster.cc:1428] Setting key 85a56d66d253e78276eac0ce65f92944
I20260926 09:22:24.314182 29629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: No bootstrap required, opened a new log
I20260926 09:22:24.315899 29629 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } }
I20260926 09:22:24.316041 29629 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:24.316066 29629 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7ed5905f194d2683c222f60d056a61, State: Initialized, Role: FOLLOWER
I20260926 09:22:24.316246 29629 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } }
I20260926 09:22:24.316334 29629 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:24.316370 29629 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:24.316418 29629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:24.317715 29629 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } }
I20260926 09:22:24.317837 29629 leader_election.cc:302] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa7ed5905f194d2683c222f60d056a61; no voters:
I20260926 09:22:24.318012 29629 leader_election.cc:288] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:24.318164 29634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:24.318305 29629 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:24.318720 29634 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [term 1 LEADER]: Becoming Leader. State: Replica: fa7ed5905f194d2683c222f60d056a61, State: Running, Role: LEADER
I20260926 09:22:24.318877 29634 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } }
I20260926 09:22:24.319252 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7ed5905f194d2683c222f60d056a61. Latest consensus state: current_term: 1 leader_uuid: "fa7ed5905f194d2683c222f60d056a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } } }
I20260926 09:22:24.319324 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:24.319625 29643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:24.320158 29643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:24.320135 29635 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa7ed5905f194d2683c222f60d056a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7ed5905f194d2683c222f60d056a61" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33849 } } }
I20260926 09:22:24.320302 29635 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61 [sys.catalog]: This master's current role is: LEADER
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:24.323477 29643 catalog_manager.cc:1347] Generated new cluster ID: 97056c9810d34d95b220b7b0bd821859
I20260926 09:22:24.323536 29643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:24.326946 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.314364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.24.235.126:33849)
0926 09:22:24.314606 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.314621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.314647 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:22:24.315891 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.315899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.315970 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.316146 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:24.316952 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.318202 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.319470 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.320225 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.320339 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.323460 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:24.323464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:24.323464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:24.323477 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:24.325325 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.325710 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.325713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.325713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.325819 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.326028 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.326029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.326029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.326312 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:24.326487 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.326581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.326736 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260926 09:22:24.346505 29643 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:24.347071 29643 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:24.354715 29643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: Generated new TSK 0
I20260926 09:22:24.354949 29643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:24.385366 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:24.481173 29655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:24.481388 29655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:24.481413 29655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:24.482555 29655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:24.482592 29655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:24.482620 29655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:24.484097 29655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:24.484289 29655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:24.484473 29655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:24.489545 29661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.489535 29660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.489535 29663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.489792 29655 server_base.cc:1034] running on GCE node
I20260926 09:22:24.490098 29655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:24.490348 29655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:24.491603 29655 hybrid_clock.cc:648] HybridClock initialized: now 1790414544491590 us; error 39 us; skew 500 ppm
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:24.494546 29655 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:24.494966 29655 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:24.495038 29655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:24.495190 29655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:24.496279 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "1c73011c180d421da8da6784bfa4c629"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "45bf5d0e0fe3c402cf8f1bb6bd0e4e69"
server_key_iv: "4dc7b42a8cd7e116b1aa65ffffb809bb"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.496665 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "1c73011c180d421da8da6784bfa4c629"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "45bf5d0e0fe3c402cf8f1bb6bd0e4e69"
server_key_iv: "4dc7b42a8cd7e116b1aa65ffffb809bb"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.498128 29655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:24.498975 29669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.499207 29655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:24.499318 29655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "1c73011c180d421da8da6784bfa4c629"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "45bf5d0e0fe3c402cf8f1bb6bd0e4e69"
server_key_iv: "4dc7b42a8cd7e116b1aa65ffffb809bb"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.499392 29655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:24.510871 29655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:24.513361 29655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:24.513540 29655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:24.513800 29655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:24.514122 29655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:24.514176 29655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.514213 29655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:24.514257 29655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.520582 29655 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:36451
I20260926 09:22:24.520650 29782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:36451 every 8 connection(s)
I20260926 09:22:24.520967 29655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:24.521487 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29655
I20260926 09:22:24.521595 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:24.521811 25517 external_mini_cluster.cc:1428] Setting key 6f95772425c9ee28e5a5319c97246443
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:22:24.531359 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.521801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55407 (local address 127.24.235.126:33849)
0926 09:22:24.521862 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.521864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.522217 (+ 353us) server_negotiation.cc:407] Connection header received
0926 09:22:24.522739 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.522740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.522756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.522813 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:24.523842 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.524375 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.525513 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.525757 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.525838 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.528132 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:24.528134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:24.528134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:24.528145 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:24.529895 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.530341 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.530343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.530343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.530406 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.530670 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.530671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.530672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.530800 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:24.530873 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.531171 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.531264 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:22:24.531582 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.521800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33849 (local address 127.24.235.65:55407)
0926 09:22:24.522006 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.522061 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.522624 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.522886 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.522890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.523207 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:24.523719 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.523724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.524486 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.524487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.525421 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.525423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.525846 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.525847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.526007 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.526622 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:24.526630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:24.528022 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:24.530004 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.530013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.530015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.530256 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.530478 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.530479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.530480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.530599 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.530878 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:24.530884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:24.531033 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.531148 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.531320 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260926 09:22:24.532087 29783 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33849
I20260926 09:22:24.532191 29783 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:24.532352 29783 heartbeater.cc:511] Master 127.24.235.126:33849 requested a full tablet report, sending...
I20260926 09:22:24.532922 29593 ts_manager.cc:194] Registered new tserver with Master: 1c73011c180d421da8da6784bfa4c629 (127.24.235.65:36451)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
I20260926 09:22:24.534003 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:55407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:24.552158 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:24.634789 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:24.635008 29788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:24.635035 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:24.636162 29788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:24.636202 29788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:24.636229 29788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:24.637804 29788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:24.638005 29788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:24.638190 29788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:24.643365 29793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.643422 29794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.643476 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.643522 29788 server_base.cc:1034] running on GCE node
I20260926 09:22:24.643785 29788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:24.644042 29788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:24.645285 29788 hybrid_clock.cc:648] HybridClock initialized: now 1790414544645280 us; error 38 us; skew 500 ppm
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:24.647840 29788 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:24.648279 29788 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:24.648356 29788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:24.648521 29788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:24.649730 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "adff31f7ca874669acd72725781e5a02"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "78457b4c31dc646a40e125c6b37736c8"
server_key_iv: "5760822f881f67fceed387ad868bb19c"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.650097 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "adff31f7ca874669acd72725781e5a02"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "78457b4c31dc646a40e125c6b37736c8"
server_key_iv: "5760822f881f67fceed387ad868bb19c"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.651433 29788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:24.652307 29802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.652705 29788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:24.652820 29788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "adff31f7ca874669acd72725781e5a02"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "78457b4c31dc646a40e125c6b37736c8"
server_key_iv: "5760822f881f67fceed387ad868bb19c"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.652899 29788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:24.664022 29788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:24.666005 29788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:24.666170 29788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:24.666400 29788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:24.666721 29788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:24.666775 29788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.666815 29788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:24.666844 29788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.672868 29788 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:39563
I20260926 09:22:24.672935 29915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:39563 every 8 connection(s)
I20260926 09:22:24.673269 29788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:24.677505 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29788
I20260926 09:22:24.677647 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:24.677851 25517 external_mini_cluster.cc:1428] Setting key 526f51661bf64e406acb0fec995d1ce2
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:24.683936 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.674131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:38323 (local address 127.24.235.126:33849)
0926 09:22:24.674203 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.674206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.674393 (+ 187us) server_negotiation.cc:407] Connection header received
0926 09:22:24.674932 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.674934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.674944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.675001 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:24.675918 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.676432 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.677626 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.677937 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.678054 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.680640 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:24.680642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:24.680643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:24.680651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:24.682241 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.682799 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.682801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.682801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.682868 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.683203 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.683205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.683205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.683325 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:24.683392 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.683780 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.683868 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:22:24.684245 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.674082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33849 (local address 127.24.235.66:38323)
0926 09:22:24.674267 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.674307 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.674815 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.675056 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.675059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.675316 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:24.675819 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.675823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.676623 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.676624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.677510 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.677512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.678066 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.678067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.678244 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.678849 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:24.678857 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:24.680539 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:24.682328 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.682335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.682337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.682704 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.682948 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.682949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.682949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.683105 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.683408 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:24.683415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:24.683614 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.683751 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.683944 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260926 09:22:24.684850 29916 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33849
I20260926 09:22:24.684947 29916 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:24.685139 29916 heartbeater.cc:511] Master 127.24.235.126:33849 requested a full tablet report, sending...
I20260926 09:22:24.685559 29593 ts_manager.cc:194] Registered new tserver with Master: adff31f7ca874669acd72725781e5a02 (127.24.235.66:39563)
I20260926 09:22:24.686120 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:38323
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:24.709333 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:24.791954 29921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:24.792184 29921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:24.792210 29921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:24.793418 29921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:24.793466 29921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:24.793500 29921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:24.795022 29921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:24.795224 29921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:24.795413 29921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:24.800873 29927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:24.800886 29926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.800987 29921 server_base.cc:1034] running on GCE node
W20260926 09:22:24.800794 29929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:24.801311 29921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:24.801628 29921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:24.802814 29921 hybrid_clock.cc:648] HybridClock initialized: now 1790414544802805 us; error 40 us; skew 500 ppm
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:24.805766 29921 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:24.806254 29921 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:24.806310 29921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:24.806461 29921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:24.807698 29921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "7d8f9006746042f6a934d6d6db04dff5"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "14b927db165e003481c0f5c7d95b0a6e"
server_key_iv: "2178634fbc262a8bcbd8c226c80f2501"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.808120 29921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "7d8f9006746042f6a934d6d6db04dff5"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "14b927db165e003481c0f5c7d95b0a6e"
server_key_iv: "2178634fbc262a8bcbd8c226c80f2501"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.809787 29921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:24.810662 29935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.810864 29921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:24.810966 29921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "7d8f9006746042f6a934d6d6db04dff5"
format_stamp: "Formatted at 2026-09-26 09:22:24 on dist-test-slave-7zlw"
server_key: "14b927db165e003481c0f5c7d95b0a6e"
server_key_iv: "2178634fbc262a8bcbd8c226c80f2501"
server_key_version: "encryptionkey@0"
I20260926 09:22:24.811049 29921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:24.825790 29921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:24.828354 29921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:24.828527 29921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:24.828816 29921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:24.829200 29921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:24.829236 29921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.829260 29921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:24.829308 29921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:24.835740 29921 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:43019
I20260926 09:22:24.835796 30048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:43019 every 8 connection(s)
I20260926 09:22:24.836104 29921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:24 dist-test-slave-7zlw krb5kdc[29549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414544, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:24.844807 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 29921
I20260926 09:22:24.844913 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:24.845146 25517 external_mini_cluster.cc:1428] Setting key 3e930df13c742a1eabeadfedf3712044
I20260926 09:22:24.846055 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.836900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:52931 (local address 127.24.235.126:33849)
0926 09:22:24.836968 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.836970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.837311 (+ 341us) server_negotiation.cc:407] Connection header received
0926 09:22:24.837817 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.837818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.837829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.837884 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:24.838833 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.839342 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.840367 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.840611 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.840676 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.842873 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:24.842874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:24.842875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:24.842882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:24.844365 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.844792 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.844793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.844794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.844857 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.845200 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.845201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.845202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.845359 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:24.845416 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.845843 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.845955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:22:24.846210 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.836884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33849 (local address 127.24.235.67:52931)
0926 09:22:24.837156 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.837208 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.837711 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.837940 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.837943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.838203 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:24.838717 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.838722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.839437 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.839437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.840277 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.840279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.840708 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.840709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.840812 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.841351 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:24.841360 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:24.842771 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:24.844446 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.844451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.844453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.844711 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.844939 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:24.844940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:24.844940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:24.845122 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:24.845430 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:24.845437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:24.845627 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.845765 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.845973 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260926 09:22:24.846603 30049 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33849
I20260926 09:22:24.846688 30049 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:24.846866 30049 heartbeater.cc:511] Master 127.24.235.126:33849 requested a full tablet report, sending...
I20260926 09:22:24.847259 29593 ts_manager.cc:194] Registered new tserver with Master: 7d8f9006746042f6a934d6d6db04dff5 (127.24.235.67:43019)
I20260926 09:22:24.847597 29593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:52931
I20260926 09:22:24.857646 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:24.865660 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.858982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.24.235.126:33849)
0926 09:22:24.859059 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.859061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.859155 (+ 94us) server_negotiation.cc:407] Connection header received
0926 09:22:24.859237 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.859238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.859249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.859321 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:24.859902 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.860397 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.861197 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.861397 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.861488 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.862703 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:24.862704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:24.862704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:24.862712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:24.864233 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.864688 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.864689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.864689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.864757 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:24.864967 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:24.864968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:24.864968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:24.865193 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:24.865306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.865451 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.865546 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:22:24.867714 29593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:24.869474 29593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:24.877455 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.24.235.67:43019)
0926 09:22:24.872943 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.872949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.872963 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:22:24.873041 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.873044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.873063 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.873157 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:24.873364 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.874747 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.876186 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.876942 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.877050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.877116 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.877338 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.877377 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260926 09:22:24.877444 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43019 (local address 127.0.0.1:44922)
0926 09:22:24.872550 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.872601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.872695 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.873179 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.873182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.873184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:24.873270 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.873273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.874874 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.874875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.876058 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.876060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.877075 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.877077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.877192 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.877210 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.877294 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.877354 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260926 09:22:24.877790 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:36451 (local address 127.0.0.1:38508)
0926 09:22:24.873153 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.873162 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.873210 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.873539 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.873540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.873542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:24.873599 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.873601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.874575 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.874576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.876057 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.876060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.877561 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.877562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.877625 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.877629 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.877690 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.877729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":64,"threads_started":1}
I20260926 09:22:24.878053 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.24.235.65:36451)
0926 09:22:24.873005 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.873037 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.873274 (+ 237us) server_negotiation.cc:407] Connection header received
0926 09:22:24.873330 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.873333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.873350 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.873430 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:24.873687 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.874443 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.876183 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.876942 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.877050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.877116 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.877901 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.877960 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260926 09:22:24.878394 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:39563 (local address 127.0.0.1:42788)
0926 09:22:24.872971 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.872983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.873071 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.873428 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.873429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.873431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:24.873473 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.873474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.876174 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.876175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.877079 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.877081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.878135 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.878136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.878204 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.878209 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.878285 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.878326 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":100,"threads_started":1}
I20260926 09:22:24.878871 29717 tablet_service.cc:1467] Processing CreateTablet for tablet dc33d9d701434725bc4c8238b95164ad (DEFAULT_TABLE table=test-table [id=8017cb56cf424cd2826bc06b19cf5b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:24.878831 29983 tablet_service.cc:1467] Processing CreateTablet for tablet dc33d9d701434725bc4c8238b95164ad (DEFAULT_TABLE table=test-table [id=8017cb56cf424cd2826bc06b19cf5b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:24.879257 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33d9d701434725bc4c8238b95164ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:24.879267 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33d9d701434725bc4c8238b95164ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:24.881127 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.872560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.24.235.66:39563)
0926 09:22:24.873078 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.873081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.873113 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:22:24.873153 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.873156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.873172 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.873331 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:24.873549 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.874756 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.877184 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.878027 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.878131 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.878191 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.878322 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.878371 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":82,"threads_started":1}
I20260926 09:22:24.882344 30067 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Bootstrap starting.
I20260926 09:22:24.883071 30067 tablet_bootstrap.cc:654] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:24.882946 29850 tablet_service.cc:1467] Processing CreateTablet for tablet dc33d9d701434725bc4c8238b95164ad (DEFAULT_TABLE table=test-table [id=8017cb56cf424cd2826bc06b19cf5b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:24.883388 29850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc33d9d701434725bc4c8238b95164ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:24.883782 30067 log.cc:824] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:24.884157 30066 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: Bootstrap starting.
I20260926 09:22:24.884654 30067 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: No bootstrap required, opened a new log
I20260926 09:22:24.884734 30067 ts_tablet_manager.cc:1397] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:22:24.884984 30066 tablet_bootstrap.cc:654] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:24.885622 30066 log.cc:824] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:24.886519 30066 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: No bootstrap required, opened a new log
I20260926 09:22:24.886600 30066 ts_tablet_manager.cc:1397] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:22:24.886719 30067 raft_consensus.cc:348] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.886883 30067 raft_consensus.cc:374] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:24.886929 30067 raft_consensus.cc:729] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8f9006746042f6a934d6d6db04dff5, State: Initialized, Role: FOLLOWER
I20260926 09:22:24.887081 30067 consensus_queue.cc:260] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.887205 30070 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Bootstrap starting.
I20260926 09:22:24.887424 30067 ts_tablet_manager.cc:1428] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:24.887594 30049 heartbeater.cc:503] Master 127.24.235.126:33849 was elected leader, sending a full tablet report...
I20260926 09:22:24.888120 30070 tablet_bootstrap.cc:654] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:24.889235 30070 log.cc:824] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:24.889739 30066 raft_consensus.cc:348] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.889930 30066 raft_consensus.cc:374] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:24.889954 30066 raft_consensus.cc:729] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c73011c180d421da8da6784bfa4c629, State: Initialized, Role: FOLLOWER
I20260926 09:22:24.890106 30066 consensus_queue.cc:260] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.890524 29783 heartbeater.cc:503] Master 127.24.235.126:33849 was elected leader, sending a full tablet report...
I20260926 09:22:24.890544 30066 ts_tablet_manager.cc:1428] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260926 09:22:24.891467 30070 tablet_bootstrap.cc:492] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: No bootstrap required, opened a new log
I20260926 09:22:24.891551 30070 ts_tablet_manager.cc:1397] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 09:22:24.893623 30070 raft_consensus.cc:348] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.893778 30070 raft_consensus.cc:374] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:24.893801 30070 raft_consensus.cc:729] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adff31f7ca874669acd72725781e5a02, State: Initialized, Role: FOLLOWER
I20260926 09:22:24.893951 30070 consensus_queue.cc:260] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.894249 30070 ts_tablet_manager.cc:1428] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 09:22:24.894330 29916 heartbeater.cc:503] Master 127.24.235.126:33849 was elected leader, sending a full tablet report...
W20260926 09:22:24.923911 29917 tablet.cc:2369] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:24.951620 30071 raft_consensus.cc:482] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:24.951795 30071 raft_consensus.cc:504] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.952189 30071 leader_election.cc:288] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adff31f7ca874669acd72725781e5a02 (127.24.235.66:39563), 1c73011c180d421da8da6784bfa4c629 (127.24.235.65:36451)
I20260926 09:22:24.955442 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.952689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:40535 (local address 127.24.235.65:36451)
0926 09:22:24.952738 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.952740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.952918 (+ 178us) server_negotiation.cc:407] Connection header received
0926 09:22:24.952955 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.952956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.952975 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.953047 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:24.953223 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.953684 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.954658 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.955086 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.955142 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.955149 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.955336 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.955371 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:24.955473 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.952587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:36451 (local address 127.24.235.67:40535)
0926 09:22:24.952791 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.952805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.952860 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.953068 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.953069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.953071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:24.953150 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.953151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.953776 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.953776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.954567 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.954569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.955178 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.955179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.955256 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.955261 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.955316 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.955374 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":152,"threads_started":1}
I20260926 09:22:24.955700 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.952452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:39563 (local address 127.24.235.67:43627)
0926 09:22:24.952528 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.952545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:24.952636 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:24.952781 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:24.952782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:24.952784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:24.952996 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.953000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.953705 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.953706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.954670 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:24.954672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.955338 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.955340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:24.955519 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.955523 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:24.955598 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:22:24.955639 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":88}
I20260926 09:22:24.955754 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.952562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:43627 (local address 127.24.235.66:39563)
0926 09:22:24.952624 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.952627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.952637 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:24.952708 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.952709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.952719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.952773 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:24.953090 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.953618 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.954757 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.955233 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.955310 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.955320 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.955621 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.955681 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:24.956130 29870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33d9d701434725bc4c8238b95164ad" candidate_uuid: "7d8f9006746042f6a934d6d6db04dff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adff31f7ca874669acd72725781e5a02" is_pre_election: true
I20260926 09:22:24.956216 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33d9d701434725bc4c8238b95164ad" candidate_uuid: "7d8f9006746042f6a934d6d6db04dff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c73011c180d421da8da6784bfa4c629" is_pre_election: true
I20260926 09:22:24.956310 29870 raft_consensus.cc:2454] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8f9006746042f6a934d6d6db04dff5 in term 0.
I20260926 09:22:24.956346 29737 raft_consensus.cc:2454] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d8f9006746042f6a934d6d6db04dff5 in term 0.
I20260926 09:22:24.956552 29938 leader_election.cc:302] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c73011c180d421da8da6784bfa4c629, 7d8f9006746042f6a934d6d6db04dff5; no voters:
I20260926 09:22:24.956686 30071 raft_consensus.cc:2781] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:24.956738 30071 raft_consensus.cc:482] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:24.956771 30071 raft_consensus.cc:3037] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:24.957598 30071 raft_consensus.cc:504] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.957752 30071 leader_election.cc:288] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [CANDIDATE]: Term 1 election: Requested vote from peers adff31f7ca874669acd72725781e5a02 (127.24.235.66:39563), 1c73011c180d421da8da6784bfa4c629 (127.24.235.65:36451)
I20260926 09:22:24.957911 29870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33d9d701434725bc4c8238b95164ad" candidate_uuid: "7d8f9006746042f6a934d6d6db04dff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adff31f7ca874669acd72725781e5a02"
I20260926 09:22:24.957957 29737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc33d9d701434725bc4c8238b95164ad" candidate_uuid: "7d8f9006746042f6a934d6d6db04dff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c73011c180d421da8da6784bfa4c629"
I20260926 09:22:24.958019 29870 raft_consensus.cc:3037] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:24.958017 29737 raft_consensus.cc:3037] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:24.958781 29737 raft_consensus.cc:2454] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8f9006746042f6a934d6d6db04dff5 in term 1.
I20260926 09:22:24.958918 29870 raft_consensus.cc:2454] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d8f9006746042f6a934d6d6db04dff5 in term 1.
I20260926 09:22:24.958954 29938 leader_election.cc:302] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c73011c180d421da8da6784bfa4c629, 7d8f9006746042f6a934d6d6db04dff5; no voters:
I20260926 09:22:24.959066 30071 raft_consensus.cc:2781] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:24.959369 30071 raft_consensus.cc:686] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 1 LEADER]: Becoming Leader. State: Replica: 7d8f9006746042f6a934d6d6db04dff5, State: Running, Role: LEADER
I20260926 09:22:24.959571 30071 consensus_queue.cc:237] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } }
I20260926 09:22:24.960520 29592 catalog_manager.cc:5495] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d8f9006746042f6a934d6d6db04dff5 (127.24.235.67). New cstate: current_term: 1 leader_uuid: "7d8f9006746042f6a934d6d6db04dff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d8f9006746042f6a934d6d6db04dff5" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:24.970748 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.968233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.24.235.67:43019)
0926 09:22:24.968292 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:24.968294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:24.968309 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:22:24.968338 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:24.968339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:24.968354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:24.968427 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:24.968605 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.969129 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.969910 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:24.970128 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:24.970206 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:24.970389 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:24.970485 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:24.970580 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:24.970624 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 09:22:24.970645 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:22:24.972762 29737 raft_consensus.cc:1261] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 1 FOLLOWER]: Refusing update from remote peer 7d8f9006746042f6a934d6d6db04dff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:24.972779 29870 raft_consensus.cc:1261] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 1 FOLLOWER]: Refusing update from remote peer 7d8f9006746042f6a934d6d6db04dff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:24.973179 30071 consensus_queue.cc:1035] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adff31f7ca874669acd72725781e5a02" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:24.973313 30076 consensus_queue.cc:1035] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c73011c180d421da8da6784bfa4c629" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:24.975950 30079 mvcc.cc:205] Tried to move back new op lower bound from 7333537976205283328 to 7333537976155394048. Current Snapshot: MvccSnapshot[applied={T|T < 7333537976205283328}]
I20260926 09:22:24.976259 30081 mvcc.cc:205] Tried to move back new op lower bound from 7333537976205283328 to 7333537976155394048. Current Snapshot: MvccSnapshot[applied={T|T < 7333537976205283328}]
I20260926 09:22:24.976570 30080 mvcc.cc:205] Tried to move back new op lower bound from 7333537976205283328 to 7333537976155394048. Current Snapshot: MvccSnapshot[applied={T|T < 7333537976205283328}]
I20260926 09:22:24.979686 29592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:24.979801 29592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:24.980727 29592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa7ed5905f194d2683c222f60d056a61: Sending DeleteTablet for 3 replicas of tablet dc33d9d701434725bc4c8238b95164ad
I20260926 09:22:24.981190 29717 tablet_service.cc:1514] Processing DeleteTablet for tablet dc33d9d701434725bc4c8238b95164ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:24 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:38508
I20260926 09:22:24.981269 29983 tablet_service.cc:1514] Processing DeleteTablet for tablet dc33d9d701434725bc4c8238b95164ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:24 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:44922
I20260926 09:22:24.981328 29850 tablet_service.cc:1514] Processing DeleteTablet for tablet dc33d9d701434725bc4c8238b95164ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:24 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42788
I20260926 09:22:24.981467 30090 tablet_replica.cc:330] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: stopping tablet replica
I20260926 09:22:24.981582 30090 raft_consensus.cc:2229] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:24.981729 30090 raft_consensus.cc:2258] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:24.981606 30089 tablet_replica.cc:330] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629: stopping tablet replica
I20260926 09:22:24.981856 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29655
I20260926 09:22:24.982009 30089 raft_consensus.cc:2229] T dc33d9d701434725bc4c8238b95164ad P 1c73011c180d421da8da6784bfa4c629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:24.981884 30091 tablet_replica.cc:330] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: stopping tablet replica
I20260926 09:22:24.982227 30091 raft_consensus.cc:2229] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:24.982244 30090 ts_tablet_manager.cc:1905] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:24.982466 30091 raft_consensus.cc:2258] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:24.983011 30091 ts_tablet_manager.cc:1905] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:24.984418 30090 ts_tablet_manager.cc:1918] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:24.984498 30090 log.cc:1196] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/dc33d9d701434725bc4c8238b95164ad
I20260926 09:22:24.984918 30090 ts_tablet_manager.cc:1939] T dc33d9d701434725bc4c8238b95164ad P 7d8f9006746042f6a934d6d6db04dff5: Deleting consensus metadata
I20260926 09:22:24.984929 30091 ts_tablet_manager.cc:1918] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:24.985133 30091 log.cc:1196] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/dc33d9d701434725bc4c8238b95164ad
I20260926 09:22:24.985424 29580 catalog_manager.cc:4841] TS 7d8f9006746042f6a934d6d6db04dff5 (127.24.235.67:43019): tablet dc33d9d701434725bc4c8238b95164ad (table test-table [id=8017cb56cf424cd2826bc06b19cf5b25]) successfully deleted
I20260926 09:22:24.985469 30091 ts_tablet_manager.cc:1939] T dc33d9d701434725bc4c8238b95164ad P adff31f7ca874669acd72725781e5a02: Deleting consensus metadata
I20260926 09:22:24.985956 29579 catalog_manager.cc:4841] TS adff31f7ca874669acd72725781e5a02 (127.24.235.66:39563): tablet dc33d9d701434725bc4c8238b95164ad (table test-table [id=8017cb56cf424cd2826bc06b19cf5b25]) successfully deleted
W20260926 09:22:24.988404 29579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:36451 (error 108)
I20260926 09:22:24.988941 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:24.988806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:36451 (local address 127.0.0.1:38516)
0926 09:22:24.988867 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:24.988913 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:22:24.989166 29579 catalog_manager.cc:4584] TS 1c73011c180d421da8da6784bfa4c629 (127.24.235.65:36451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc33d9d701434725bc4c8238b95164ad: Network error: Client connection negotiation failed: client connection to 127.24.235.65:36451: connect: Connection refused (error 111)
I20260926 09:22:24.990617 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29788
I20260926 09:22:25.000517 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29921
I20260926 09:22:25.008148 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 29562
2026-09-26T09:22:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1992 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T09:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:25Z Disabled control of system clock
I20260926 09:22:25.038568 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37509
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:22:25.121861 30096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:25.122094 30096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:25.122117 30096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:25.123281 30096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:25.123328 30096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:25.123342 30096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:25.123355 30096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:25.123368 30096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:25.124691 30096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:25.124903 30096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:25.125174 30096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:25.130769 30102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.130810 30104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.130939 30096 server_base.cc:1034] running on GCE node
W20260926 09:22:25.130777 30101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.131309 30096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:25.131668 30096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:25.132841 30096 hybrid_clock.cc:648] HybridClock initialized: now 1790414545132826 us; error 42 us; skew 500 ppm
I20260926 09:22:25.134464 30096 webserver.cc:460] Webserver started at http://127.24.235.126:43965/ using document root <none> and password file <none>
I20260926 09:22:25.134755 30096 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:25.134809 30096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:25.134977 30096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:25.136180 30096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "2300f35584f3494885cc109d0cb5faf3"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "25f04ef4ed646ad0220399cecc523246"
server_key_iv: "3a1e70140795bc5124c88191a2a53d8b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.136591 30096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "2300f35584f3494885cc109d0cb5faf3"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "25f04ef4ed646ad0220399cecc523246"
server_key_iv: "3a1e70140795bc5124c88191a2a53d8b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.138166 30096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:25.139146 30110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.139339 30096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:25.139503 30096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "2300f35584f3494885cc109d0cb5faf3"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "25f04ef4ed646ad0220399cecc523246"
server_key_iv: "3a1e70140795bc5124c88191a2a53d8b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.139598 30096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:25.143261 30096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:25.143567 30096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:25.143726 30096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:25.148175 30096 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37509
I20260926 09:22:25.148236 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37509 every 8 connection(s)
I20260926 09:22:25.148631 30096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:25.149406 30163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:25.152308 30163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: Bootstrap starting.
I20260926 09:22:25.152884 30163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:25.153210 30163 log.cc:824] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:25.153574 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30096
I20260926 09:22:25.153687 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:25.153877 25517 external_mini_cluster.cc:1428] Setting key 0fda64dec74e40fa0829b3e4e678186c
I20260926 09:22:25.154034 30163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: No bootstrap required, opened a new log
I20260926 09:22:25.156335 30163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } }
I20260926 09:22:25.156522 30163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:25.156562 30163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2300f35584f3494885cc109d0cb5faf3, State: Initialized, Role: FOLLOWER
I20260926 09:22:25.156718 30163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } }
I20260926 09:22:25.156785 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.154447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.24.235.126:37509)
0926 09:22:25.154681 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.154691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.154728 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:22:25.155427 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.155437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.155489 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.155654 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.156337 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.156340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.156341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.156346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.156396 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.156410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.156491 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.156537 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.156597 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260926 09:22:25.156809 30163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:25.156845 30163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:25.156891 30163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:25.158058 30163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } }
I20260926 09:22:25.158154 30163 leader_election.cc:302] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2300f35584f3494885cc109d0cb5faf3; no voters:
I20260926 09:22:25.158326 30163 leader_election.cc:288] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:25.158418 30168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:25.158593 30168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [term 1 LEADER]: Becoming Leader. State: Replica: 2300f35584f3494885cc109d0cb5faf3, State: Running, Role: LEADER
I20260926 09:22:25.158632 30163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:25.158702 30168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } }
I20260926 09:22:25.159139 30170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2300f35584f3494885cc109d0cb5faf3. Latest consensus state: current_term: 1 leader_uuid: "2300f35584f3494885cc109d0cb5faf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } } }
I20260926 09:22:25.159125 30169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2300f35584f3494885cc109d0cb5faf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2300f35584f3494885cc109d0cb5faf3" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37509 } } }
I20260926 09:22:25.159472 30170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:25.159475 30169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:25.159776 30177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:25.160531 30177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:25.162163 30177 catalog_manager.cc:1347] Generated new cluster ID: fc8cdb7a6bbd486a9b09efbee6c5821f
I20260926 09:22:25.162217 30177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:25.177475 30177 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:25.177685 30177 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:25.180751 30177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: Generated new TSK 0
I20260926 09:22:25.180953 30177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:25.182153 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--builtin_ntp_servers=127.24.235.84:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:22:25.282509 30187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:25.282732 30187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:25.282753 30187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:25.283913 30187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:25.283952 30187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:25.283978 30187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:25.285353 30187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:25.285626 30187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:25.285936 30187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:25.291214 30195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.291265 30192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.291381 30193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.291877 30187 server_base.cc:1034] running on GCE node
I20260926 09:22:25.292120 30187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:25.292436 30187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:25.293638 30187 hybrid_clock.cc:648] HybridClock initialized: now 1790414545293619 us; error 36 us; skew 500 ppm
I20260926 09:22:25.295254 30187 webserver.cc:460] Webserver started at http://127.24.235.65:38337/ using document root <none> and password file <none>
I20260926 09:22:25.295555 30187 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:25.295642 30187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:25.295785 30187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:25.297356 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "1be47c7d3dcd4527b3599915d1aa36fe"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "173d820263c3527df8cd6b313e3ec5dc"
server_key_iv: "e7c1155a616cb68e1834c595ce5ee8b8"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.297828 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "1be47c7d3dcd4527b3599915d1aa36fe"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "173d820263c3527df8cd6b313e3ec5dc"
server_key_iv: "e7c1155a616cb68e1834c595ce5ee8b8"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.299731 30187 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:25.300765 30201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.300983 30187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:25.301115 30187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "1be47c7d3dcd4527b3599915d1aa36fe"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "173d820263c3527df8cd6b313e3ec5dc"
server_key_iv: "e7c1155a616cb68e1834c595ce5ee8b8"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.301178 30187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:25.304482 30187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:25.304710 30187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:25.304806 30187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:25.304951 30187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:25.305294 30187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:25.305354 30187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.305399 30187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:25.305428 30187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.311015 30187 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:39269
I20260926 09:22:25.311108 30314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:39269 every 8 connection(s)
I20260926 09:22:25.311463 30187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:25.315469 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.312273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:42603 (local address 127.24.235.126:37509)
0926 09:22:25.312345 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.312347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.312622 (+ 275us) server_negotiation.cc:407] Connection header received
0926 09:22:25.313343 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.313345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.313352 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.313387 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.315194 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.315196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.315196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.315200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.315216 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.315220 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.315277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.315400 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.315412 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:22:25.315848 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.312246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37509 (local address 127.24.235.65:42603)
0926 09:22:25.312454 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.312502 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.313206 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.313610 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.313614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.313625 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.314991 (+ 1366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.314991 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.315033 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.315071 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.315305 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.315312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.315316 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.315403 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.315473 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260926 09:22:25.316514 30315 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37509
I20260926 09:22:25.316624 30315 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:25.316780 30315 heartbeater.cc:511] Master 127.24.235.126:37509 requested a full tablet report, sending...
I20260926 09:22:25.317366 30127 ts_manager.cc:194] Registered new tserver with Master: 1be47c7d3dcd4527b3599915d1aa36fe (127.24.235.65:39269)
I20260926 09:22:25.318389 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30187
I20260926 09:22:25.318473 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:25.318693 25517 external_mini_cluster.cc:1428] Setting key 3d17a82849e97857d2e7411b1414eff6
I20260926 09:22:25.319747 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--builtin_ntp_servers=127.24.235.84:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:22:25.411648 30318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:25.411868 30318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:25.411891 30318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:25.413236 30318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:25.413285 30318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:25.413316 30318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:25.414834 30318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:25.415293 30318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:25.415712 30318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:25.421356 30323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.421356 30326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.421384 30324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.421664 30318 server_base.cc:1034] running on GCE node
I20260926 09:22:25.421919 30318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:25.422220 30318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:25.423408 30318 hybrid_clock.cc:648] HybridClock initialized: now 1790414545423389 us; error 35 us; skew 500 ppm
I20260926 09:22:25.424856 30318 webserver.cc:460] Webserver started at http://127.24.235.66:39971/ using document root <none> and password file <none>
I20260926 09:22:25.425192 30318 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:25.425267 30318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:25.425412 30318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:25.426599 30318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "bf3832f2e0be4e178826bb1316ea01ab"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "e54928872ef915715199c10597b4b205"
server_key_iv: "2e78ed521a04f1ce7e40b7320db826da"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.426990 30318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "bf3832f2e0be4e178826bb1316ea01ab"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "e54928872ef915715199c10597b4b205"
server_key_iv: "2e78ed521a04f1ce7e40b7320db826da"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.428372 30318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:25.429337 30332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.429668 30318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:25.429776 30318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "bf3832f2e0be4e178826bb1316ea01ab"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "e54928872ef915715199c10597b4b205"
server_key_iv: "2e78ed521a04f1ce7e40b7320db826da"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.429864 30318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:25.433383 30318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:25.433665 30318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:25.433808 30318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:25.434054 30318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:25.434397 30318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:25.434451 30318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.434492 30318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:25.434520 30318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.440508 30318 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:40337
I20260926 09:22:25.440546 30445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:40337 every 8 connection(s)
I20260926 09:22:25.440903 30318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:25.444360 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.441755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:42769 (local address 127.24.235.126:37509)
0926 09:22:25.441853 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.441855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.442078 (+ 223us) server_negotiation.cc:407] Connection header received
0926 09:22:25.442541 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.442542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.442549 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.442618 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.444120 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.444121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.444122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.444126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.444143 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.444148 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.444213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.444285 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.444299 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 09:22:25.444587 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.441716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37509 (local address 127.24.235.66:42769)
0926 09:22:25.441908 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.441942 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.442413 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.442726 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.442730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.442739 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.443949 (+ 1210us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.443949 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.443980 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.444014 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.444213 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.444216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.444218 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.444272 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.444323 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260926 09:22:25.445092 30446 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37509
I20260926 09:22:25.445211 30446 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:25.445379 30446 heartbeater.cc:511] Master 127.24.235.126:37509 requested a full tablet report, sending...
I20260926 09:22:25.445851 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30318
I20260926 09:22:25.445854 30127 ts_manager.cc:194] Registered new tserver with Master: bf3832f2e0be4e178826bb1316ea01ab (127.24.235.66:40337)
I20260926 09:22:25.445920 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:25.446229 25517 external_mini_cluster.cc:1428] Setting key cf6302ad04d33f5b7bb3eb2fbd9e982f
I20260926 09:22:25.447253 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--builtin_ntp_servers=127.24.235.84:33845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 09:22:25.534268 30449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:25.534509 30449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:25.534546 30449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:25.535792 30449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:25.535863 30449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:25.535907 30449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:25.537345 30449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:33845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:25.537618 30449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:25.537930 30449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:25.543350 30449 server_base.cc:1034] running on GCE node
W20260926 09:22:25.543344 30454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.543368 30457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.543365 30455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.543865 30449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:25.544152 30449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:25.545322 30449 hybrid_clock.cc:648] HybridClock initialized: now 1790414545545327 us; error 47 us; skew 500 ppm
I20260926 09:22:25.546692 30449 webserver.cc:460] Webserver started at http://127.24.235.67:37111/ using document root <none> and password file <none>
I20260926 09:22:25.546958 30449 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:25.547029 30449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:25.547188 30449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:25.548302 30449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "94d836e7fc24401ca52fc1b31ab39661"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f2610591595818860cfb67ad64126119"
server_key_iv: "e06d47ec71782cece71f5563b7cfd15b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.548704 30449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "94d836e7fc24401ca52fc1b31ab39661"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f2610591595818860cfb67ad64126119"
server_key_iv: "e06d47ec71782cece71f5563b7cfd15b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.550207 30449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:25.551081 30463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.551271 30449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:25.551378 30449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "94d836e7fc24401ca52fc1b31ab39661"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f2610591595818860cfb67ad64126119"
server_key_iv: "e06d47ec71782cece71f5563b7cfd15b"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.551452 30449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:25.555027 30449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:25.555294 30449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:25.555438 30449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:25.555645 30449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:25.555950 30449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:25.556002 30449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.556041 30449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:25.556067 30449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.562193 30449 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:46315
I20260926 09:22:25.562278 30576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:46315 every 8 connection(s)
I20260926 09:22:25.562608 30449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:25.565737 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.563452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37263 (local address 127.24.235.126:37509)
0926 09:22:25.563528 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.563530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.563729 (+ 199us) server_negotiation.cc:407] Connection header received
0926 09:22:25.564172 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.564173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.564178 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.564235 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.565464 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.565465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.565466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.565470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.565486 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.565490 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.565565 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.565669 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.565683 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:22:25.565907 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.563418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37509 (local address 127.24.235.67:37263)
0926 09:22:25.563592 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.563641 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.564091 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.564294 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.564298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.564306 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.565266 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.565266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.565316 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.565347 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.565569 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.565573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.565576 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.565640 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.565689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 09:22:25.566401 30577 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37509
I20260926 09:22:25.566490 30577 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:25.566604 30577 heartbeater.cc:511] Master 127.24.235.126:37509 requested a full tablet report, sending...
I20260926 09:22:25.566951 30127 ts_manager.cc:194] Registered new tserver with Master: 94d836e7fc24401ca52fc1b31ab39661 (127.24.235.67:46315)
I20260926 09:22:25.572690 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30449
I20260926 09:22:25.572875 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:25.573175 25517 external_mini_cluster.cc:1428] Setting key d84b2fbb737232ac26d14d874e384b33
I20260926 09:22:25.574219 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:25.576761 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.575699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.24.235.126:37509)
0926 09:22:25.575745 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.575748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.575802 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:22:25.575877 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.575878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.575884 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.575930 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.576533 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.576535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.576535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.576539 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.576554 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.576558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.576607 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.576665 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.576681 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:25.578789 30127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:25.580489 30127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:25.585546 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.583021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:46315 (local address 127.0.0.1:48990)
0926 09:22:25.583309 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.583319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.583350 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.584102 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.584102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.584105 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.584825 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.584826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.584846 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.584875 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.585322 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.585330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.585333 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.585417 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.585436 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":73,"threads_started":1}
I20260926 09:22:25.586283 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.583249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.24.235.67:46315)
0926 09:22:25.583441 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.583444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.583452 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:25.583472 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.583474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.583481 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.583566 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.585093 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.585101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.585102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.585131 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.585185 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.585214 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.585288 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.586146 (+ 858us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.586167 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260926 09:22:25.587482 30511 tablet_service.cc:1467] Processing CreateTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 (DEFAULT_TABLE table=test-table [id=9b813962d0c5486f8dbdfb2a04797534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:25.588012 30511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c6e7ebec2b47a587316f5c651bb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:25.588135 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.582808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:39269 (local address 127.0.0.1:59650)
0926 09:22:25.583197 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.583206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.583241 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.583551 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.583557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.583565 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.586743 (+ 3178us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.586744 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.586752 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.586765 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.587979 (+ 1214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.587981 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.587982 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.588018 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.588035 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260926 09:22:25.588449 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.582920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.24.235.65:39269)
0926 09:22:25.583243 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.583247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.583260 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:25.583321 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.583324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.583332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.583536 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.587800 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.587805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.587805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.587810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.587853 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.587870 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.588283 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.588315 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.588336 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260926 09:22:25.589291 30249 tablet_service.cc:1467] Processing CreateTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 (DEFAULT_TABLE table=test-table [id=9b813962d0c5486f8dbdfb2a04797534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:25.589700 30249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c6e7ebec2b47a587316f5c651bb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:25.591437 30591 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Bootstrap starting.
I20260926 09:22:25.592185 30591 tablet_bootstrap.cc:654] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:25.592597 30591 log.cc:824] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:25.592697 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.582808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40337 (local address 127.0.0.1:59864)
0926 09:22:25.582994 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.583036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.583091 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.591762 (+ 8671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.591764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.591767 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.592293 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.592294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.592303 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.592318 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.592540 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.592541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.592542 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.592602 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.592618 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260926 09:22:25.592726 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.583040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.24.235.66:40337)
0926 09:22:25.583206 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.583211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.583224 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:25.583256 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.583259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.583271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.591966 (+ 8695us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.592428 (+ 462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.592432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.592433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.592436 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.592463 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.592477 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.592541 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.592607 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.592623 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260926 09:22:25.593397 30380 tablet_service.cc:1467] Processing CreateTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 (DEFAULT_TABLE table=test-table [id=9b813962d0c5486f8dbdfb2a04797534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:25.593777 30380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c6e7ebec2b47a587316f5c651bb9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:25.593798 30591 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: No bootstrap required, opened a new log
I20260926 09:22:25.593864 30591 ts_tablet_manager.cc:1397] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:25.595650 30593 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: Bootstrap starting.
I20260926 09:22:25.597077 30593 tablet_bootstrap.cc:654] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:25.597189 30591 raft_consensus.cc:348] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.597443 30591 raft_consensus.cc:374] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:25.597484 30591 raft_consensus.cc:729] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d836e7fc24401ca52fc1b31ab39661, State: Initialized, Role: FOLLOWER
I20260926 09:22:25.597595 30593 log.cc:824] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:25.597646 30591 consensus_queue.cc:260] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.597924 30591 ts_tablet_manager.cc:1428] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:22:25.598079 30577 heartbeater.cc:503] Master 127.24.235.126:37509 was elected leader, sending a full tablet report...
I20260926 09:22:25.598372 30596 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Bootstrap starting.
I20260926 09:22:25.599119 30596 tablet_bootstrap.cc:654] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:25.599363 30593 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: No bootstrap required, opened a new log
I20260926 09:22:25.599429 30593 ts_tablet_manager.cc:1397] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 09:22:25.599457 30596 log.cc:824] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:25.600452 30596 tablet_bootstrap.cc:492] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: No bootstrap required, opened a new log
I20260926 09:22:25.600517 30596 ts_tablet_manager.cc:1397] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:25.601955 30593 raft_consensus.cc:348] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.602130 30593 raft_consensus.cc:374] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:25.602188 30593 raft_consensus.cc:729] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be47c7d3dcd4527b3599915d1aa36fe, State: Initialized, Role: FOLLOWER
I20260926 09:22:25.602329 30593 consensus_queue.cc:260] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.602712 30593 ts_tablet_manager.cc:1428] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260926 09:22:25.602752 30315 heartbeater.cc:503] Master 127.24.235.126:37509 was elected leader, sending a full tablet report...
I20260926 09:22:25.603821 30596 raft_consensus.cc:348] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.604009 30596 raft_consensus.cc:374] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:25.604045 30596 raft_consensus.cc:729] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3832f2e0be4e178826bb1316ea01ab, State: Initialized, Role: FOLLOWER
I20260926 09:22:25.604177 30596 consensus_queue.cc:260] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.604523 30596 ts_tablet_manager.cc:1428] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 09:22:25.604629 30446 heartbeater.cc:503] Master 127.24.235.126:37509 was elected leader, sending a full tablet report...
I20260926 09:22:25.653968 30598 raft_consensus.cc:482] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:25.654158 30598 raft_consensus.cc:504] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.654589 30598 leader_election.cc:288] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf3832f2e0be4e178826bb1316ea01ab (127.24.235.66:40337), 94d836e7fc24401ca52fc1b31ab39661 (127.24.235.67:46315)
I20260926 09:22:25.656112 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.654892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:57071 (local address 127.24.235.66:40337)
0926 09:22:25.654939 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.654941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.654950 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:25.654984 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.654985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.654992 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.655063 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.655764 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.655766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.655767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.655772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.655800 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.655806 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.655879 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.656014 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.656029 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:25.656123 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.654810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40337 (local address 127.24.235.65:57071)
0926 09:22:25.654867 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.654880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.654924 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.655131 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.655143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.655147 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.655641 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.655642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.655651 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.655667 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.655901 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.655902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.655903 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.656009 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.656026 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:22:25.656582 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c6e7ebec2b47a587316f5c651bb9f2" candidate_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3832f2e0be4e178826bb1316ea01ab" is_pre_election: true
I20260926 09:22:25.656772 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.655139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:35437 (local address 127.24.235.67:46315)
0926 09:22:25.655199 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.655201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.655209 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:25.655242 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.655243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.655248 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.655339 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.656490 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.656492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.656493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.656499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.656517 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.656522 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.656556 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.656691 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.656707 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:25.656775 30400 raft_consensus.cc:2454] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be47c7d3dcd4527b3599915d1aa36fe in term 0.
I20260926 09:22:25.657068 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.654936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:46315 (local address 127.24.235.65:35437)
0926 09:22:25.655076 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.655086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.655129 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.655350 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.655351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.655353 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:25.656371 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:25.656371 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:25.656378 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:25.656391 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:25.656617 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:25.656618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:25.656618 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 09:22:25.656860 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 09:22:25.656875 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260926 09:22:25.657084 30204 leader_election.cc:302] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be47c7d3dcd4527b3599915d1aa36fe, bf3832f2e0be4e178826bb1316ea01ab; no voters:
I20260926 09:22:25.657248 30598 raft_consensus.cc:2781] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:25.657337 30598 raft_consensus.cc:482] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:25.657367 30598 raft_consensus.cc:3037] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:25.657454 30531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c6e7ebec2b47a587316f5c651bb9f2" candidate_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d836e7fc24401ca52fc1b31ab39661" is_pre_election: true
I20260926 09:22:25.657639 30531 raft_consensus.cc:2454] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1be47c7d3dcd4527b3599915d1aa36fe in term 0.
I20260926 09:22:25.658509 30598 raft_consensus.cc:504] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.658666 30598 leader_election.cc:288] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [CANDIDATE]: Term 1 election: Requested vote from peers bf3832f2e0be4e178826bb1316ea01ab (127.24.235.66:40337), 94d836e7fc24401ca52fc1b31ab39661 (127.24.235.67:46315)
I20260926 09:22:25.658923 30531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c6e7ebec2b47a587316f5c651bb9f2" candidate_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d836e7fc24401ca52fc1b31ab39661"
I20260926 09:22:25.659026 30531 raft_consensus.cc:3037] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:25.658895 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c6e7ebec2b47a587316f5c651bb9f2" candidate_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3832f2e0be4e178826bb1316ea01ab"
I20260926 09:22:25.659121 30400 raft_consensus.cc:3037] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:25.660379 30400 raft_consensus.cc:2454] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be47c7d3dcd4527b3599915d1aa36fe in term 1.
I20260926 09:22:25.660557 30204 leader_election.cc:302] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be47c7d3dcd4527b3599915d1aa36fe, bf3832f2e0be4e178826bb1316ea01ab; no voters:
I20260926 09:22:25.660857 30531 raft_consensus.cc:2454] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be47c7d3dcd4527b3599915d1aa36fe in term 1.
I20260926 09:22:25.661288 30598 raft_consensus.cc:2781] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:25.661475 30598 raft_consensus.cc:686] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [term 1 LEADER]: Becoming Leader. State: Replica: 1be47c7d3dcd4527b3599915d1aa36fe, State: Running, Role: LEADER
I20260926 09:22:25.661579 30598 consensus_queue.cc:237] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } }
I20260926 09:22:25.662377 30127 catalog_manager.cc:5495] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1be47c7d3dcd4527b3599915d1aa36fe (127.24.235.65). New cstate: current_term: 1 leader_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be47c7d3dcd4527b3599915d1aa36fe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:25.685351 30587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.684203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.24.235.65:39269)
0926 09:22:25.684276 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.684278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.684288 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:25.684333 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.684334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.684352 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.684423 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.685109 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.685110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.685111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.685115 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.685130 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.685134 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.685193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.685258 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.685272 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:22:25.687364 30400 raft_consensus.cc:1261] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 1 FOLLOWER]: Refusing update from remote peer 1be47c7d3dcd4527b3599915d1aa36fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:25.687389 30531 raft_consensus.cc:1261] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 1 FOLLOWER]: Refusing update from remote peer 1be47c7d3dcd4527b3599915d1aa36fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:25.687898 30601 consensus_queue.cc:1035] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d836e7fc24401ca52fc1b31ab39661" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:25.688061 30598 consensus_queue.cc:1035] T 67c6e7ebec2b47a587316f5c651bb9f2 P 1be47c7d3dcd4527b3599915d1aa36fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3832f2e0be4e178826bb1316ea01ab" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:25.691085 30604 mvcc.cc:205] Tried to move back new op lower bound from 7333537979132583936 to 7333537979030454272. Current Snapshot: MvccSnapshot[applied={T|T < 7333537979132583936}]
W20260926 09:22:25.691437 30447 tablet.cc:2369] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:25.691598 30605 mvcc.cc:205] Tried to move back new op lower bound from 7333537979132583936 to 7333537979030454272. Current Snapshot: MvccSnapshot[applied={T|T < 7333537979132583936}]
I20260926 09:22:25.692459 30607 mvcc.cc:205] Tried to move back new op lower bound from 7333537979132583936 to 7333537979030454272. Current Snapshot: MvccSnapshot[applied={T|T < 7333537979132583936}]
I20260926 09:22:25.693833 30126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:25.693946 30126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:25.694767 30126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2300f35584f3494885cc109d0cb5faf3: Sending DeleteTablet for 3 replicas of tablet 67c6e7ebec2b47a587316f5c651bb9f2
I20260926 09:22:25.695164 30249 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:25 UTC) from {username='slave'} at 127.0.0.1:59650
I20260926 09:22:25.695294 30380 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:25 UTC) from {username='slave'} at 127.0.0.1:59864
I20260926 09:22:25.695482 30511 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c6e7ebec2b47a587316f5c651bb9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:25 UTC) from {username='slave'} at 127.0.0.1:48990
I20260926 09:22:25.695528 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30187
I20260926 09:22:25.696166 30615 tablet_replica.cc:330] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: stopping tablet replica
I20260926 09:22:25.696188 30614 tablet_replica.cc:330] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: stopping tablet replica
I20260926 09:22:25.696267 30615 raft_consensus.cc:2229] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:25.696283 30614 raft_consensus.cc:2229] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:25.696367 30615 raft_consensus.cc:2258] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:25.696380 30614 raft_consensus.cc:2258] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:25.696862 30615 ts_tablet_manager.cc:1905] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:25.696873 30614 ts_tablet_manager.cc:1905] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:25.699409 30614 ts_tablet_manager.cc:1918] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:25.699487 30614 log.cc:1196] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/67c6e7ebec2b47a587316f5c651bb9f2
I20260926 09:22:25.699501 30615 ts_tablet_manager.cc:1918] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:25.699553 30615 log.cc:1196] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/67c6e7ebec2b47a587316f5c651bb9f2
I20260926 09:22:25.699787 30615 ts_tablet_manager.cc:1939] T 67c6e7ebec2b47a587316f5c651bb9f2 P 94d836e7fc24401ca52fc1b31ab39661: Deleting consensus metadata
I20260926 09:22:25.699836 30614 ts_tablet_manager.cc:1939] T 67c6e7ebec2b47a587316f5c651bb9f2 P bf3832f2e0be4e178826bb1316ea01ab: Deleting consensus metadata
I20260926 09:22:25.700171 30113 catalog_manager.cc:4841] TS 94d836e7fc24401ca52fc1b31ab39661 (127.24.235.67:46315): tablet 67c6e7ebec2b47a587316f5c651bb9f2 (table test-table [id=9b813962d0c5486f8dbdfb2a04797534]) successfully deleted
I20260926 09:22:25.700263 30113 catalog_manager.cc:4841] TS bf3832f2e0be4e178826bb1316ea01ab (127.24.235.66:40337): tablet 67c6e7ebec2b47a587316f5c651bb9f2 (table test-table [id=9b813962d0c5486f8dbdfb2a04797534]) successfully deleted
W20260926 09:22:25.702445 30114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:39269 (error 108)
I20260926 09:22:25.702994 30586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.702838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:39269 (local address 127.0.0.1:59680)
0926 09:22:25.702908 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.702958 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:22:25.703035 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30318
W20260926 09:22:25.703739 30114 catalog_manager.cc:4584] TS 1be47c7d3dcd4527b3599915d1aa36fe (127.24.235.65:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67c6e7ebec2b47a587316f5c651bb9f2: Network error: Client connection negotiation failed: client connection to 127.24.235.65:39269: connect: Connection refused (error 111)
I20260926 09:22:25.708478 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30449
I20260926 09:22:25.713274 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30096
2026-09-26T09:22:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (703 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 09:22:25.733645 25517 mini_oidc.cc:152] Starting JWKS server
I20260926 09:22:25.734777 25517 webserver.cc:460] Webserver started at https://127.0.0.1:33403/ using document root <none> and password file <none>
I20260926 09:22:25.734908 25517 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:22:25.735108 25517 webserver.cc:460] Webserver started at http://127.0.0.1:45861/ using document root <none> and password file <none>
2026-09-26T09:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:25Z Disabled control of system clock
I20260926 09:22:25.743587 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36787
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36787
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33403/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/testchainca.cert with env {}
W20260926 09:22:25.827105 30622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:25.827347 30622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:25.827371 30622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:25.828681 30622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:25.828730 30622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:25.828743 30622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:25.828756 30622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:25.828768 30622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:22:25.828781 30622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:22:25.828792 30622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33403/jwks.json
I20260926 09:22:25.830513 30622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33403/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:25.830746 30622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:25.830910 30622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:25.835994 30628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.836018 30630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:25.836001 30627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:25.836360 30622 server_base.cc:1034] running on GCE node
I20260926 09:22:25.836685 30622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:25.837054 30622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:25.838280 30622 hybrid_clock.cc:648] HybridClock initialized: now 1790414545838277 us; error 43 us; skew 500 ppm
I20260926 09:22:25.839696 30622 webserver.cc:460] Webserver started at http://127.24.235.126:45613/ using document root <none> and password file <none>
I20260926 09:22:25.839941 30622 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:25.840023 30622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:25.840178 30622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:25.841353 30622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "c510fee96f054493a94558ab17f18db1"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f7d76990acb56246c7a4c4949d08f47f"
server_key_iv: "e99d055f5294f0155c174380f0c00982"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.841712 30622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "c510fee96f054493a94558ab17f18db1"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f7d76990acb56246c7a4c4949d08f47f"
server_key_iv: "e99d055f5294f0155c174380f0c00982"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.843050 30622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:25.843755 30636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:25.843930 30622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:25.844041 30622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "c510fee96f054493a94558ab17f18db1"
format_stamp: "Formatted at 2026-09-26 09:22:25 on dist-test-slave-7zlw"
server_key: "f7d76990acb56246c7a4c4949d08f47f"
server_key_iv: "e99d055f5294f0155c174380f0c00982"
server_key_version: "encryptionkey@0"
I20260926 09:22:25.844115 30622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:25.870671 30622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:25.871057 30622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:25.871224 30622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:25.875783 30622 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36787
I20260926 09:22:25.875841 30688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36787 every 8 connection(s)
I20260926 09:22:25.876216 30622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:25.877267 30689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:25.878510 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30622
I20260926 09:22:25.878604 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:25.878826 25517 external_mini_cluster.cc:1428] Setting key ddfd43ba869f486ced8eeebeb722de55
I20260926 09:22:25.880362 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: Bootstrap starting.
I20260926 09:22:25.881187 30689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:25.881513 30689 log.cc:824] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:25.882287 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: No bootstrap required, opened a new log
I20260926 09:22:25.884986 30689 raft_consensus.cc:348] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } }
I20260926 09:22:25.885231 30689 raft_consensus.cc:374] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:25.885288 30689 raft_consensus.cc:729] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c510fee96f054493a94558ab17f18db1, State: Initialized, Role: FOLLOWER
I20260926 09:22:25.885466 30689 consensus_queue.cc:260] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } }
I20260926 09:22:25.885540 30689 raft_consensus.cc:388] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:25.885658 30689 raft_consensus.cc:482] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:25.885720 30689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:25.885914 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.879435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.24.235.126:36787)
0926 09:22:25.879664 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.879674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.879714 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:22:25.880251 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.880260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.880343 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.880505 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:25.880806 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.882170 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.883065 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.883723 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.883872 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:25.884764 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:25.884768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:25.884769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:25.885077 (+ 308us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:25.885124 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:25.885140 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:25.885226 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:25.885372 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 09:22:25.885472 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260926 09:22:25.886610 30689 raft_consensus.cc:504] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } }
I20260926 09:22:25.886749 30689 leader_election.cc:302] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c510fee96f054493a94558ab17f18db1; no voters:
I20260926 09:22:25.886951 30689 leader_election.cc:288] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:25.886983 30694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:25.887042 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:22:25.887109 30694 raft_consensus.cc:686] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [term 1 LEADER]: Becoming Leader. State: Replica: c510fee96f054493a94558ab17f18db1, State: Running, Role: LEADER
I20260926 09:22:25.887202 30694 consensus_queue.cc:237] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } }
I20260926 09:22:25.887252 30689 sys_catalog.cc:564] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:25.887580 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c510fee96f054493a94558ab17f18db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } } }
I20260926 09:22:25.887661 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:25.887962 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c510fee96f054493a94558ab17f18db1. Latest consensus state: current_term: 1 leader_uuid: "c510fee96f054493a94558ab17f18db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c510fee96f054493a94558ab17f18db1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36787 } } }
I20260926 09:22:25.888058 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1 [sys.catalog]: This master's current role is: LEADER
W20260926 09:22:25.889523 30709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:22:25.889573 30709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:25.889701 30710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:25.889925 30710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:25.891644 30710 catalog_manager.cc:1347] Generated new cluster ID: bfc76a7ee4c8478994fd5df20ac6e298
I20260926 09:22:25.891710 30710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:25.904517 30710 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:25.905375 30710 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:25.917619 30710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c510fee96f054493a94558ab17f18db1: Generated new TSK 0
I20260926 09:22:25.917869 30710 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 09:22:25.946224 30719 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 09:22:25.946359 30719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 09:22:25.946658 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:25.930269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.24.235.126:36787)
0926 09:22:25.930323 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:25.930326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:25.930445 (+ 119us) server_negotiation.cc:407] Connection header received
0926 09:22:25.930514 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:25.930514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:25.930527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:25.930625 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:22:25.930780 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.931668 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.932447 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.932797 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.932874 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:25.933084 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:22:25.946397 (+ 13313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:22:25.946572 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 09:22:25.946743 30692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 09:22:25.947198 30718 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:25.930178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36787 (local address 127.0.0.1:42686)
0926 09:22:25.930364 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:25.930374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:25.930434 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:25.930638 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:25.930638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:25.930640 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:22:25.930703 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:25.930704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.931766 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.931767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:25.932351 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:25.932352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:25.932881 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:25.932882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:25.932935 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:25.932953 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:22:25.946645 (+ 13692us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 09:22:25.946793 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.235.126:36787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.235.126:36787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 09:22:25.947744 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30622
2026-09-26T09:22:25Z chronyd exiting
I20260926 09:22:25.984227 25517 test_util.cc:175] -----------------------------------------------
I20260926 09:22:25.984336 25517 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790414522268193-25517-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 09:22:25.985208 25517 mini_oidc.cc:152] Starting JWKS server
I20260926 09:22:25.985917 25517 webserver.cc:460] Webserver started at https://127.0.0.1:39443/ using document root <none> and password file <none>
I20260926 09:22:25.985985 25517 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:22:25.986133 25517 webserver.cc:460] Webserver started at http://127.0.0.1:33937/ using document root <none> and password file <none>
2026-09-26T09:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:25Z Disabled control of system clock
I20260926 09:22:25.994498 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46791
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:46791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/testchainca.cert with env {}
W20260926 09:22:26.076623 30725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:26.076829 30725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:26.076853 30725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:26.078032 30725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:26.078074 30725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:26.078088 30725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:26.078100 30725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:26.078114 30725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:22:26.078125 30725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:22:26.078136 30725 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39443/jwks.json
I20260926 09:22:26.079664 30725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:46791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:26.079885 30725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:26.080050 30725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:26.084790 30731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.084851 30733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.084851 30730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:26.085496 30725 server_base.cc:1034] running on GCE node
I20260926 09:22:26.085789 30725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:26.086081 30725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:26.087252 30725 hybrid_clock.cc:648] HybridClock initialized: now 1790414546087253 us; error 43 us; skew 500 ppm
I20260926 09:22:26.094192 30725 webserver.cc:460] Webserver started at http://127.24.235.126:36465/ using document root <none> and password file <none>
I20260926 09:22:26.094477 30725 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:26.094552 30725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:26.094722 30725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:26.095854 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "dd15f4ddf7bc4592bed9a37f48edaf70"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "75043cafe6c123052d6aa8490ed3ec66"
server_key_iv: "7fdd252bf10265c850c7195a239f6d02"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.096231 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "dd15f4ddf7bc4592bed9a37f48edaf70"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "75043cafe6c123052d6aa8490ed3ec66"
server_key_iv: "7fdd252bf10265c850c7195a239f6d02"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.097649 30725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:26.098407 30739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:26.098578 30725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:26.098683 30725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "dd15f4ddf7bc4592bed9a37f48edaf70"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "75043cafe6c123052d6aa8490ed3ec66"
server_key_iv: "7fdd252bf10265c850c7195a239f6d02"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.098764 30725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:26.116071 30725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:26.116428 30725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:26.116590 30725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:26.120848 30725 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:46791
I20260926 09:22:26.120980 30791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:46791 every 8 connection(s)
I20260926 09:22:26.121327 30725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:26.122076 30792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:26.125149 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70: Bootstrap starting.
I20260926 09:22:26.125774 30792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:26.126066 30792 log.cc:824] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:26.126927 30792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70: No bootstrap required, opened a new log
I20260926 09:22:26.128633 30792 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } }
I20260926 09:22:26.128777 30792 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:26.128799 30792 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd15f4ddf7bc4592bed9a37f48edaf70, State: Initialized, Role: FOLLOWER
I20260926 09:22:26.128971 30792 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } }
I20260926 09:22:26.129097 30792 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:26.129154 30792 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:26.129211 30792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:26.129417 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30725
I20260926 09:22:26.129536 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:26.129747 25517 external_mini_cluster.cc:1428] Setting key 5f2e1685cceb092f0740826324f9c64c
I20260926 09:22:26.129930 30792 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } }
I20260926 09:22:26.130055 30792 leader_election.cc:302] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd15f4ddf7bc4592bed9a37f48edaf70; no voters:
I20260926 09:22:26.130235 30792 leader_election.cc:288] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:26.130333 30795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:26.130517 30792 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:26.130609 30795 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [term 1 LEADER]: Becoming Leader. State: Replica: dd15f4ddf7bc4592bed9a37f48edaf70, State: Running, Role: LEADER
I20260926 09:22:26.130716 30795 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } }
I20260926 09:22:26.131132 30795 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd15f4ddf7bc4592bed9a37f48edaf70. Latest consensus state: current_term: 1 leader_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } } }
I20260926 09:22:26.131139 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd15f4ddf7bc4592bed9a37f48edaf70" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46791 } } }
I20260926 09:22:26.131233 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.131376 30795 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.131498 30806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:26.132040 30806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:26.133817 30806 catalog_manager.cc:1347] Generated new cluster ID: 74885cc0325f40b8adf620a07bc80169
I20260926 09:22:26.133864 30806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:26.137198 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.130303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.24.235.126:46791)
0926 09:22:26.131559 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.131570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.131605 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:22:26.131724 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.131732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.131814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.132224 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:26.132425 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.133864 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.134999 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.135616 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.135729 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.136468 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:26.136472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:26.136472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:26.136680 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:26.136721 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:26.136735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:26.136798 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:26.136888 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 09:22:26.136965 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":82,"threads_started":1}
I20260926 09:22:26.138170 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:22:26.140553 30806 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:26.141007 30806 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:26.146903 30806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd15f4ddf7bc4592bed9a37f48edaf70: Generated new TSK 0
I20260926 09:22:26.147125 30806 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 09:22:26.177201 30822 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 09:22:26.177299 30822 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 09:22:26.177616 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.154676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.24.235.126:46791)
0926 09:22:26.154741 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.154743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.154840 (+ 97us) server_negotiation.cc:407] Connection header received
0926 09:22:26.154931 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.154932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.154943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.155006 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:22:26.155158 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.155684 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.156805 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.156999 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.157114 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.157319 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:22:26.177397 (+ 20078us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:22:26.177558 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:22:26.177717 30798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 09:22:26.177798 30821 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:26.154621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46791 (local address 127.0.0.1:58920)
0926 09:22:26.154776 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:26.154789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:26.154859 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:26.155026 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:26.155028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:26.155030 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:22:26.155089 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.155091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.155782 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.155783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.156721 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.156724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.157141 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.157143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.157213 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.157219 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:22:26.177542 (+ 20323us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 09:22:26.177678 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.235.126:46791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260926 09:22:26.178224 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30725
2026-09-26T09:22:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 09:22:26.216045 25517 mini_oidc.cc:152] Starting JWKS server
I20260926 09:22:26.216725 25517 webserver.cc:460] Webserver started at https://127.0.0.1:34123/ using document root <none> and password file <none>
I20260926 09:22:26.216768 25517 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 09:22:26.216908 25517 webserver.cc:460] Webserver started at http://127.0.0.1:37403/ using document root <none> and password file <none>
2026-09-26T09:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:26Z Disabled control of system clock
I20260926 09:22:26.225258 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36465
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:22:26.308125 30828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:26.308370 30828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:26.308393 30828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:26.309654 30828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:26.309705 30828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:26.309718 30828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:26.309731 30828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:26.309743 30828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:22:26.309756 30828 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:22:26.309767 30828 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34123/jwks.json
I20260926 09:22:26.311160 30828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:26.311383 30828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:26.311642 30828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:26.316960 30836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.316947 30834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.316947 30833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:26.317293 30828 server_base.cc:1034] running on GCE node
I20260926 09:22:26.317519 30828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:26.317843 30828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:26.319010 30828 hybrid_clock.cc:648] HybridClock initialized: now 1790414546318984 us; error 46 us; skew 500 ppm
I20260926 09:22:26.320528 30828 webserver.cc:460] Webserver started at http://127.24.235.126:33453/ using document root <none> and password file <none>
I20260926 09:22:26.320861 30828 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:26.320948 30828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:26.321144 30828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:26.322333 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "1673521328894dbfaf81d032d34e7666"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "f044f962a3bcb087caeb475d06fdf4c8"
server_key_iv: "0b29eca230f2ae1df5b55f2a20bd4ed0"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.322741 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "1673521328894dbfaf81d032d34e7666"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "f044f962a3bcb087caeb475d06fdf4c8"
server_key_iv: "0b29eca230f2ae1df5b55f2a20bd4ed0"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.324185 30828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:26.325215 30842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:26.325407 30828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:26.325523 30828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "1673521328894dbfaf81d032d34e7666"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "f044f962a3bcb087caeb475d06fdf4c8"
server_key_iv: "0b29eca230f2ae1df5b55f2a20bd4ed0"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.325661 30828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:26.334318 30828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:26.334681 30828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:26.334846 30828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:26.339329 30828 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36465
I20260926 09:22:26.339370 30894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36465 every 8 connection(s)
I20260926 09:22:26.339792 30828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:26.340029 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30828
I20260926 09:22:26.340164 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:26.340402 25517 external_mini_cluster.cc:1428] Setting key da6ed34889969aade0c16d772cd7dee2
I20260926 09:22:26.340689 30895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:26.343766 30895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666: Bootstrap starting.
I20260926 09:22:26.344478 30895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:26.344772 30895 log.cc:824] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:26.345499 30895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666: No bootstrap required, opened a new log
I20260926 09:22:26.346863 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.341382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.24.235.126:36465)
0926 09:22:26.341598 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.341606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.341632 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:22:26.342271 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.342279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.342349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.342516 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:26.342713 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.343680 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.344645 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.345227 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.345384 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.346166 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:26.346170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:26.346171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:26.346342 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:26.346377 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:26.346388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:26.346474 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:26.346556 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 09:22:26.346700 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260926 09:22:26.347779 30895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } }
I20260926 09:22:26.347939 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:22:26.347944 30895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:26.347988 30895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1673521328894dbfaf81d032d34e7666, State: Initialized, Role: FOLLOWER
I20260926 09:22:26.348150 30895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } }
I20260926 09:22:26.348234 30895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:26.348256 30895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:26.348327 30895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:26.348976 30895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } }
I20260926 09:22:26.349143 30895 leader_election.cc:302] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1673521328894dbfaf81d032d34e7666; no voters:
I20260926 09:22:26.349339 30895 leader_election.cc:288] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:26.349372 30900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:26.349496 30900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [term 1 LEADER]: Becoming Leader. State: Replica: 1673521328894dbfaf81d032d34e7666, State: Running, Role: LEADER
I20260926 09:22:26.349578 30900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } }
I20260926 09:22:26.349606 30895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:26.349984 30901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1673521328894dbfaf81d032d34e7666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } } }
I20260926 09:22:26.349992 30902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1673521328894dbfaf81d032d34e7666. Latest consensus state: current_term: 1 leader_uuid: "1673521328894dbfaf81d032d34e7666" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1673521328894dbfaf81d032d34e7666" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36465 } } }
I20260926 09:22:26.350087 30901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.350101 30902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.350332 30906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:26.350924 30906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:26.352542 30906 catalog_manager.cc:1347] Generated new cluster ID: ebdacc89c30f4cd18a8a5e633cb62df7
I20260926 09:22:26.352602 30906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:26.364262 30906 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:26.365123 30906 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:26.372382 30906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1673521328894dbfaf81d032d34e7666: Generated new TSK 0
I20260926 09:22:26.372624 30906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:26.384112 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.380161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.24.235.126:36465)
0926 09:22:26.380232 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.380235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.380360 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:22:26.380419 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.380420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.380435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.380521 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:22:26.380644 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.381477 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.382282 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.382602 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.382676 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.382882 (+ 206us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:22:26.383886 (+ 1004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:22:26.384034 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 09:22:26.384183 30898 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 09:22:26.386221 30924 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:26.380051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36465 (local address 127.0.0.1:52196)
0926 09:22:26.380283 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:26.380300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:26.380358 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:26.380527 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:26.380527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:26.380529 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:22:26.380580 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.380582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.381578 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.381579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.382191 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.382192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.382703 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.382705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.382759 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.382765 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:22:26.383996 (+ 1231us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 09:22:26.386113 (+ 2117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.235.126:36465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1899,"thread_start_us":167,"threads_started":1}
I20260926 09:22:26.386673 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30828
E20260926 09:22:26.397060 30925 webserver.cc:533] Webserver: sslize failed:
E20260926 09:22:26.397151 30925 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T09:22:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T09:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:26Z Disabled control of system clock
I20260926 09:22:26.434274 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36903
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36903
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/testchainca.cert with env {}
W20260926 09:22:26.522746 30929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:26.522967 30929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:26.522990 30929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:26.524192 30929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:26.524242 30929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:26.524256 30929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:26.524269 30929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:26.524281 30929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 09:22:26.524294 30929 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 09:22:26.524350 30929 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 09:22:26.525985 30929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:26.526253 30929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:26.526580 30929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:26.531741 30935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.531749 30934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:26.531770 30937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:26.532066 30929 server_base.cc:1034] running on GCE node
I20260926 09:22:26.532373 30929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:26.532689 30929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:26.533859 30929 hybrid_clock.cc:648] HybridClock initialized: now 1790414546533847 us; error 48 us; skew 500 ppm
I20260926 09:22:26.535171 30929 webserver.cc:460] Webserver started at http://127.24.235.126:40421/ using document root <none> and password file <none>
I20260926 09:22:26.535427 30929 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:26.535496 30929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:26.535684 30929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:26.536757 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "79e86fdda1cf41908606599422f33bcd"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "bd0b2d76b90e0f5452eec27f15fec368"
server_key_iv: "d80bc7e98eeac487ee1e0e9fcb931c54"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.537187 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "79e86fdda1cf41908606599422f33bcd"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "bd0b2d76b90e0f5452eec27f15fec368"
server_key_iv: "d80bc7e98eeac487ee1e0e9fcb931c54"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.538465 30929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:26.539302 30943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:26.539474 30929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:26.539573 30929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "79e86fdda1cf41908606599422f33bcd"
format_stamp: "Formatted at 2026-09-26 09:22:26 on dist-test-slave-7zlw"
server_key: "bd0b2d76b90e0f5452eec27f15fec368"
server_key_iv: "d80bc7e98eeac487ee1e0e9fcb931c54"
server_key_version: "encryptionkey@0"
I20260926 09:22:26.539691 30929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:26.556739 30929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:26.557169 30929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:26.557336 30929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:26.561324 30929 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36903
I20260926 09:22:26.561388 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36903 every 8 connection(s)
I20260926 09:22:26.561758 30929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:26.562753 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:26.564970 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd: Bootstrap starting.
I20260926 09:22:26.565603 30996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:26.565846 30996 log.cc:824] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:26.566520 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd: No bootstrap required, opened a new log
I20260926 09:22:26.568059 30996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } }
I20260926 09:22:26.568225 30996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:26.568248 30996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e86fdda1cf41908606599422f33bcd, State: Initialized, Role: FOLLOWER
I20260926 09:22:26.568389 30996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } }
I20260926 09:22:26.568471 30996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:26.568495 30996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:26.568539 30996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:26.569110 30996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } }
I20260926 09:22:26.569204 30996 leader_election.cc:302] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e86fdda1cf41908606599422f33bcd; no voters:
I20260926 09:22:26.569386 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 30929
I20260926 09:22:26.569443 30996 leader_election.cc:288] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:26.569481 30999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:26.569502 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:26.569619 30999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [term 1 LEADER]: Becoming Leader. State: Replica: 79e86fdda1cf41908606599422f33bcd, State: Running, Role: LEADER
I20260926 09:22:26.569733 25517 external_mini_cluster.cc:1428] Setting key 9721075c9324257e78c4e8553fd4e942
I20260926 09:22:26.569741 30996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:26.569769 30999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } }
I20260926 09:22:26.570089 30999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e86fdda1cf41908606599422f33bcd. Latest consensus state: current_term: 1 leader_uuid: "79e86fdda1cf41908606599422f33bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } } }
I20260926 09:22:26.570169 30999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.570297 31001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e86fdda1cf41908606599422f33bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e86fdda1cf41908606599422f33bcd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36903 } } }
I20260926 09:22:26.570423 31001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:26.570675 31006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:26.571241 31006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:26.574066 31006 catalog_manager.cc:1347] Generated new cluster ID: 3c8efe4665664363863402a45fb25225
I20260926 09:22:26.574126 31006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:26.576650 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.570412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.24.235.126:36903)
0926 09:22:26.570635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.570643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.570668 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:22:26.570799 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.570806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.570885 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.571032 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:26.571948 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.572931 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.574089 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.574708 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.574824 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.575776 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:26.575781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:26.575782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:26.576008 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:26.576048 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:26.576064 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:26.576122 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:26.576329 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 09:22:26.576415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260926 09:22:26.577648 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:22:26.583060 31006 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:26.583760 31006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:26.598328 31006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e86fdda1cf41908606599422f33bcd: Generated new TSK 0
I20260926 09:22:26.598650 31006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:26.696864 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:26.608358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.24.235.126:36903)
0926 09:22:26.608423 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:26.608426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:26.608511 (+ 85us) server_negotiation.cc:407] Connection header received
0926 09:22:26.608658 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:26.608659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:26.608670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:26.608749 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0926 09:22:26.608891 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.609467 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.610623 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.610921 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.610997 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.611186 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 09:22:26.696517 (+ 85331us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:22:26.696766 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 09:22:26.696949 31010 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 09:22:26.696974 31025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:26.608271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36903 (local address 127.0.0.1:59128)
0926 09:22:26.608429 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:26.608441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:26.608531 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:26.608750 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:26.608751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:26.608753 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 09:22:26.608821 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.608823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.609552 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.609553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.610533 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:26.610536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:26.610998 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:26.610999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:26.611058 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:26.611064 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 09:22:26.696728 (+ 85664us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 09:22:26.696853 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.235.126:36903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 09:22:26.697346 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 30929
2026-09-26T09:22:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (288 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:26 dist-test-slave-7zlw krb5kdc[31037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:26 dist-test-slave-7zlw krb5kdc[31037](info): set up 2 sockets
Sep 26 09:22:26 dist-test-slave-7zlw krb5kdc[31037](info): commencing operation
krb5kdc: starting...
W20260926 09:22:27.921468 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:28.049597 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41397
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:28.132102 31050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:28.132325 31050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:28.132350 31050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:28.133618 31050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:28.133668 31050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:28.133683 31050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:28.133697 31050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:28.133708 31050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:28.135243 31050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:28.135524 31050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:28.135778 31050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:28.141324 31058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.141324 31056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.141319 31055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:28.141355 31050 server_base.cc:1034] running on GCE node
I20260926 09:22:28.142861 31050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:28.143167 31050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:28.144317 31050 hybrid_clock.cc:648] HybridClock initialized: now 1790414548144305 us; error 33 us; skew 500 ppm
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.146950 31050 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:28.147467 31050 webserver.cc:460] Webserver started at http://127.24.235.126:43065/ using document root <none> and password file <none>
I20260926 09:22:28.147722 31050 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:28.147792 31050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:28.147943 31050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:28.149142 31050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "76ea384a76194a8dac876433382c9679"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "030d46e7d6d7da28d0f3452840ccca2e"
server_key_iv: "1c9bcba53d2c58ab5647bd5416422c5d"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.149519 31050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "76ea384a76194a8dac876433382c9679"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "030d46e7d6d7da28d0f3452840ccca2e"
server_key_iv: "1c9bcba53d2c58ab5647bd5416422c5d"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.150918 31050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:22:28.151783 31065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.151954 31050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:28.152065 31050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "76ea384a76194a8dac876433382c9679"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "030d46e7d6d7da28d0f3452840ccca2e"
server_key_iv: "1c9bcba53d2c58ab5647bd5416422c5d"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.152146 31050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:28.172329 31050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:28.174456 31050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:28.174638 31050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:28.178987 31050 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41397
I20260926 09:22:28.179023 31117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41397 every 8 connection(s)
I20260926 09:22:28.179401 31050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:28.180308 31118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:28.182602 31118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: Bootstrap starting.
I20260926 09:22:28.183148 31118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:28.183382 31118 log.cc:824] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:28.184055 31118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: No bootstrap required, opened a new log
I20260926 09:22:28.184996 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31050
I20260926 09:22:28.185143 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:28.185354 25517 external_mini_cluster.cc:1428] Setting key 29276ccdfcfdf002fad96f026ae6e004
I20260926 09:22:28.186293 31118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } }
I20260926 09:22:28.186465 31118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:28.186501 31118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ea384a76194a8dac876433382c9679, State: Initialized, Role: FOLLOWER
I20260926 09:22:28.186628 31118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } }
I20260926 09:22:28.186713 31118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:28.186736 31118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:28.186800 31118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:28.187891 31118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } }
I20260926 09:22:28.187999 31118 leader_election.cc:302] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ea384a76194a8dac876433382c9679; no voters:
I20260926 09:22:28.188138 31118 leader_election.cc:288] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:28.188232 31123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:28.188409 31118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:28.188488 31123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [term 1 LEADER]: Becoming Leader. State: Replica: 76ea384a76194a8dac876433382c9679, State: Running, Role: LEADER
I20260926 09:22:28.188617 31123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } }
I20260926 09:22:28.189067 31124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ea384a76194a8dac876433382c9679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } } }
I20260926 09:22:28.189147 31124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:28.189291 31125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ea384a76194a8dac876433382c9679. Latest consensus state: current_term: 1 leader_uuid: "76ea384a76194a8dac876433382c9679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ea384a76194a8dac876433382c9679" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41397 } } }
I20260926 09:22:28.189339 31125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679 [sys.catalog]: This master's current role is: LEADER
W20260926 09:22:28.190245 31139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:22:28.190296 31139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:28.190341 31132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:28.190483 31132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:28.192157 31132 catalog_manager.cc:1347] Generated new cluster ID: fa8851177547418c8eb3205ec605f987
I20260926 09:22:28.192212 31132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:28.196973 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.185889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.24.235.126:41397)
0926 09:22:28.186101 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.186111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.186155 (+ 44us) server_negotiation.cc:407] Connection header received
0926 09:22:28.186273 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.186281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.186339 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.186488 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.187280 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.188788 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.189977 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.190612 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.190730 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.193641 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.193645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.193646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.193658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.195380 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.195774 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.195780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.195780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.195893 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.196123 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.196124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.196125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.196370 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.196558 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.196636 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.196781 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260926 09:22:28.201423 31132 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:28.202010 31132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:28.208202 31132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ea384a76194a8dac876433382c9679: Generated new TSK 0
I20260926 09:22:28.208408 31132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:28.239961 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:28.325522 31144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:28.325775 31144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:28.325817 31144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:28.327044 31144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:28.327114 31144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:28.327160 31144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:28.328737 31144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:28.329077 31144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:28.329337 31144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:28.334667 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.334654 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.334647 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:28.335229 31144 server_base.cc:1034] running on GCE node
I20260926 09:22:28.335551 31144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:28.335857 31144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:28.337077 31144 hybrid_clock.cc:648] HybridClock initialized: now 1790414548337049 us; error 28 us; skew 500 ppm
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.339840 31144 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:28.340329 31144 webserver.cc:460] Webserver started at http://127.24.235.65:33527/ using document root <none> and password file <none>
I20260926 09:22:28.340585 31144 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:28.340631 31144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:28.340772 31144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:28.342170 31144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "968f66d076e04b88b19caa7c766816ba"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "a207f856043231a615c3c9c9e30c52c1"
server_key_iv: "5714b49bd9fcea87108eba85baa18af7"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.342631 31144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "968f66d076e04b88b19caa7c766816ba"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "a207f856043231a615c3c9c9e30c52c1"
server_key_iv: "5714b49bd9fcea87108eba85baa18af7"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.344035 31144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:28.345067 31159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.345253 31144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:28.345369 31144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "968f66d076e04b88b19caa7c766816ba"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "a207f856043231a615c3c9c9e30c52c1"
server_key_iv: "5714b49bd9fcea87108eba85baa18af7"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.345464 31144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:28.354846 31144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:28.357245 31144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:28.357411 31144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:28.357650 31144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:28.357967 31144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:28.358021 31144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.358064 31144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:28.358093 31144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.364411 31144 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:45121
I20260926 09:22:28.364470 31272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:45121 every 8 connection(s)
I20260926 09:22:28.364841 31144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:28.365831 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31144
I20260926 09:22:28.365969 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:28.366174 25517 external_mini_cluster.cc:1428] Setting key 882dd27c2e181b8c3fe9e3e3c92678eb
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:22:28.375350 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.365715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:53311 (local address 127.24.235.126:41397)
0926 09:22:28.365766 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.365768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.365971 (+ 203us) server_negotiation.cc:407] Connection header received
0926 09:22:28.367169 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.367171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.367186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.367241 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.367918 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.368457 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.369638 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.369879 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.369972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.372213 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.372215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.372215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.372226 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.373812 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.374253 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.374254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.374255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.374318 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.374633 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.374634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.374634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.374767 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.374846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.375156 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.375253 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:28.375547 31275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.365649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41397 (local address 127.24.235.65:53311)
0926 09:22:28.365814 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:28.365858 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:28.367060 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:28.367316 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:28.367320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:28.367599 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:28.367810 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.367815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.368562 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.368563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.369530 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.369532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.369983 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.369985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.370117 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.370697 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:28.370706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:28.372098 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:28.373917 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.373926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.373929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.374172 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.374394 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.374395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.374395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.374550 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.374846 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:28.374853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:28.375003 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 09:22:28.375117 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 09:22:28.375288 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260926 09:22:28.376060 31273 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41397
I20260926 09:22:28.376151 31273 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:28.376380 31273 heartbeater.cc:511] Master 127.24.235.126:41397 requested a full tablet report, sending...
I20260926 09:22:28.377069 31082 ts_manager.cc:194] Registered new tserver with Master: 968f66d076e04b88b19caa7c766816ba (127.24.235.65:45121)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
I20260926 09:22:28.377924 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:53311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:28.397658 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:28.480082 31278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:28.480373 31278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:28.480417 31278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:28.481783 31278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:28.481870 31278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:28.481916 31278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:28.483518 31278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:28.483821 31278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:28.484068 31278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:28.489369 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.489403 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.489369 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:28.489683 31278 server_base.cc:1034] running on GCE node
I20260926 09:22:28.489912 31278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:28.490130 31278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:28.491267 31278 hybrid_clock.cc:648] HybridClock initialized: now 1790414548491254 us; error 28 us; skew 500 ppm
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.493953 31278 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:28.494398 31278 webserver.cc:460] Webserver started at http://127.24.235.66:43967/ using document root <none> and password file <none>
I20260926 09:22:28.494630 31278 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:28.494675 31278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:28.494840 31278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:28.495922 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "c9f59d6e69bd487087257fd613aa4aac"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "74a432db46fb39be81e59117822777d6"
server_key_iv: "b499a9ce53791e646f80ea04b76c3619"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.496308 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f59d6e69bd487087257fd613aa4aac"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "74a432db46fb39be81e59117822777d6"
server_key_iv: "b499a9ce53791e646f80ea04b76c3619"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.497803 31278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:28.498601 31293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.498773 31278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.498871 31278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "c9f59d6e69bd487087257fd613aa4aac"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "74a432db46fb39be81e59117822777d6"
server_key_iv: "b499a9ce53791e646f80ea04b76c3619"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.498953 31278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:28.520303 31278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:28.522984 31278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:28.523167 31278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:28.523435 31278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:28.523787 31278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:28.523844 31278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.523880 31278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:28.523907 31278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.529819 31278 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:45443
I20260926 09:22:28.529894 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:45443 every 8 connection(s)
I20260926 09:22:28.530238 31278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:28.533236 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31278
I20260926 09:22:28.533365 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:28.533617 25517 external_mini_cluster.cc:1428] Setting key 5e8e18f16cd11394abcfbb3da80d5dfc
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:28.541517 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.531071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:32857 (local address 127.24.235.126:41397)
0926 09:22:28.531163 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.531165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.531364 (+ 199us) server_negotiation.cc:407] Connection header received
0926 09:22:28.532373 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.532375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.532391 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.532443 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.533172 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.533676 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.534829 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.535039 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.535126 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.538140 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.538143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.538143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.538153 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.539891 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.540357 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.540358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.540359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.540425 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.540728 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.540729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.540730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.540862 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.540954 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.541332 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.541423 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:22:28.541718 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.531069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41397 (local address 127.24.235.66:32857)
0926 09:22:28.531226 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:28.531264 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:28.532233 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:28.532522 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:28.532525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:28.532823 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:28.533060 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.533065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.533770 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.533771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.534740 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.534742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.535136 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.535137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.535303 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.535914 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:28.535925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:28.537957 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:28.540011 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.540017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.540019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.540271 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.540497 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.540498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.540498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.540613 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.540942 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:28.540949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:28.541179 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 09:22:28.541295 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 09:22:28.541464 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260926 09:22:28.542289 31407 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41397
I20260926 09:22:28.542397 31407 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:28.542603 31407 heartbeater.cc:511] Master 127.24.235.126:41397 requested a full tablet report, sending...
I20260926 09:22:28.543038 31082 ts_manager.cc:194] Registered new tserver with Master: c9f59d6e69bd487087257fd613aa4aac (127.24.235.66:45443)
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
I20260926 09:22:28.543483 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:32857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:28.566118 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:28.648702 31412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:28.648943 31412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:28.648981 31412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:28.650188 31412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:28.650261 31412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:28.650307 31412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:28.651844 31412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:28.652105 31412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:28.652347 31412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:28.657716 31417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.657696 31420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:28.657752 31418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:28.658164 31412 server_base.cc:1034] running on GCE node
I20260926 09:22:28.658442 31412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:28.658763 31412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:28.659945 31412 hybrid_clock.cc:648] HybridClock initialized: now 1790414548659932 us; error 30 us; skew 500 ppm
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.662810 31412 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:28.663213 31412 webserver.cc:460] Webserver started at http://127.24.235.67:44351/ using document root <none> and password file <none>
I20260926 09:22:28.663414 31412 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:28.663455 31412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:28.663542 31412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:28.664609 31412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "448de7ec5f064d35952d574c89d49418"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "b9ba30c4dfe736c40ab8644bcc4ebbc0"
server_key_iv: "473e5d91787e53482d940c126a7ac981"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.664921 31412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "448de7ec5f064d35952d574c89d49418"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "b9ba30c4dfe736c40ab8644bcc4ebbc0"
server_key_iv: "473e5d91787e53482d940c126a7ac981"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.666334 31412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:28.667125 31427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.667303 31412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:28.667394 31412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "448de7ec5f064d35952d574c89d49418"
format_stamp: "Formatted at 2026-09-26 09:22:28 on dist-test-slave-7zlw"
server_key: "b9ba30c4dfe736c40ab8644bcc4ebbc0"
server_key_iv: "473e5d91787e53482d940c126a7ac981"
server_key_version: "encryptionkey@0"
I20260926 09:22:28.667451 31412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:28.692731 31412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:28.694974 31412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:28.695096 31412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:28.695305 31412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:28.695659 31412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:28.695698 31412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.695721 31412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:28.695734 31412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:28.701855 31412 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:40501
I20260926 09:22:28.701918 31540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:40501 every 8 connection(s)
I20260926 09:22:28.702239 31412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:28.711683 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31412
I20260926 09:22:28.711781 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:28.711848 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.703006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:35501 (local address 127.24.235.126:41397)
0926 09:22:28.703076 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.703079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.703371 (+ 292us) server_negotiation.cc:407] Connection header received
0926 09:22:28.704275 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.704277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.704287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.704344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.704996 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.705541 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.706499 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.706689 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.706742 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.709113 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.709115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.709115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.709122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.710563 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.710934 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.710935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.710935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.710984 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.711204 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.711205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.711205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.711308 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.711360 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.711661 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.711720 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:22:28.711967 25517 external_mini_cluster.cc:1428] Setting key 93901aeef5cd1cee20924e61e66491ea
I20260926 09:22:28.712059 31543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.703045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41397 (local address 127.24.235.67:35501)
0926 09:22:28.703241 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:28.703280 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:28.704177 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:28.704437 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:28.704441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:28.704722 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:28.704919 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.704923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.705622 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.705622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.706421 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:28.706423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.706762 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.706763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:28.706861 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.707338 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:28.707346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:28.708928 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:28.710641 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.710648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.710651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.710865 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.711042 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:28.711042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:28.711043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:28.711149 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:28.711363 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:28.711367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:28.711519 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 09:22:28.711643 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 09:22:28.711825 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260926 09:22:28.712565 31541 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41397
I20260926 09:22:28.712682 31541 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:28.712879 31541 heartbeater.cc:511] Master 127.24.235.126:41397 requested a full tablet report, sending...
I20260926 09:22:28.713299 31082 ts_manager.cc:194] Registered new tserver with Master: 448de7ec5f064d35952d574c89d49418 (127.24.235.67:40501)
I20260926 09:22:28.713649 31082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:35501
I20260926 09:22:28.724056 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/test-admin.keytab.
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.741791 25517 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 09:22:28.743395 31548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:28.742636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.24.235.65:45121)
0926 09:22:28.742805 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.742810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.742822 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:28.742857 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.742859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.742872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.742944 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.743283 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39108: BlockingRecv error: recv got EOF from 127.0.0.1:39108 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260926 09:22:28.743610 25517 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc' of type FILE
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.744620 25517 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.745579 25517 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:28.754096 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.746089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.24.235.65:45121)
0926 09:22:28.746170 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.746172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.746182 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:28.746238 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.746239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.746250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.746296 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.746652 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.747375 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.748297 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.748642 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.748701 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.751119 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.751128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.751128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.751145 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.752735 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.753249 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.753253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.753253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.753313 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.753533 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.753533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.753533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.753680 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.753796 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.753893 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.753968 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 09:22:28.756139 31548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:28.755458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.24.235.65:45121)
0926 09:22:28.755515 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.755517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.755662 (+ 145us) server_negotiation.cc:407] Connection header received
0926 09:22:28.755726 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.755726 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.755738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.755786 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.756065 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39116: BlockingRecv error: recv got EOF from 127.0.0.1:39116 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:22:28.756223 25517 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc' of type FILE
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.757416 25517 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.758335 25517 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:28.765297 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.758868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.24.235.65:45121)
0926 09:22:28.758931 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.758933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.758974 (+ 41us) server_negotiation.cc:407] Connection header received
0926 09:22:28.759036 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.759036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.759047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.759091 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.759461 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.759948 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.760631 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.760853 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.760932 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.762972 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.762974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.762974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.762982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.764157 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.764480 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.764481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.764482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.764540 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.764775 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.764776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.764776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.764881 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.764982 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.765115 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.765196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 09:22:28.766860 31548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:28.766260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.24.235.65:45121)
0926 09:22:28.766306 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.766307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.766331 (+ 24us) server_negotiation.cc:407] Connection header received
0926 09:22:28.766391 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.766391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.766399 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.766439 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.766801 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39136: BlockingRecv error: recv got EOF from 127.0.0.1:39136 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 09:22:28.766945 25517 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790414522268193-25517-0/krb5kdc/krb5cc)
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.767923 25517 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:28.768862 25517 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.65@KRBTEST.COM
I20260926 09:22:28.776326 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:28.769644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.24.235.65:45121)
0926 09:22:28.769715 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:28.769717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:28.769730 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:28.769748 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:28.769749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:28.769756 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:28.769808 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:28.770282 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.770821 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.771589 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:28.771775 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:28.771832 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:28.773910 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:28.773912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:28.773912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:28.773918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:28.775370 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.775683 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.775684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.775684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.775732 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:28.775894 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:28.775894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:28.775894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:28.775982 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:28.776053 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:28.776133 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:22:28.776188 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 09:22:28.776937 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31144
I20260926 09:22:28.781975 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31278
I20260926 09:22:28.786648 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31412
I20260926 09:22:28.791539 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31050
2026-09-26T09:22:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1915 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31561](info): set up 2 sockets
Sep 26 09:22:28 dist-test-slave-7zlw krb5kdc[31561](info): commencing operation
krb5kdc: starting...
W20260926 09:22:29.841497 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:29 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab.
Entry for principal oryx/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab.
I20260926 09:22:29.968732 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43935
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab
--principal=oryx/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:30.055016 31574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:30.055243 31574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:30.055266 31574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:30.056385 31574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:30.056427 31574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:30.056442 31574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:30.056454 31574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:30.056466 31574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:30.058023 31574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab
--principal=oryx/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:30.058262 31574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:30.058451 31574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.063472 31579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.063488 31580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.063488 31582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.063768 31574 server_base.cc:1034] running on GCE node
I20260926 09:22:30.064043 31574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.064342 31574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.065487 31574 hybrid_clock.cc:648] HybridClock initialized: now 1790414550065469 us; error 74 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.068056 31574 init.cc:377] Logged in from keytab as oryx/127.24.235.126@KRBTEST.COM (short username oryx)
I20260926 09:22:30.068540 31574 webserver.cc:460] Webserver started at http://127.24.235.126:39969/ using document root <none> and password file <none>
I20260926 09:22:30.068778 31574 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.068848 31574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.069005 31574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:30.070140 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "e68798c8cfff45639b9b0ce71cdff09b"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "51e95d33d771babc38c5bb111e268e83"
server_key_iv: "60da1b314259723ec2366c54838e450e"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.070504 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "e68798c8cfff45639b9b0ce71cdff09b"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "51e95d33d771babc38c5bb111e268e83"
server_key_iv: "60da1b314259723ec2366c54838e450e"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.071904 31574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:30.072670 31589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.072849 31574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:30.072952 31574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "e68798c8cfff45639b9b0ce71cdff09b"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "51e95d33d771babc38c5bb111e268e83"
server_key_iv: "60da1b314259723ec2366c54838e450e"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.073055 31574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.093585 31574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.096009 31574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:30.096204 31574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.100461 31574 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:43935
I20260926 09:22:30.100517 31641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:43935 every 8 connection(s)
I20260926 09:22:30.100898 31574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:30.101912 31642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:30.104187 31642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Bootstrap starting.
I20260926 09:22:30.104391 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31574
I20260926 09:22:30.104504 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:30.104722 25517 external_mini_cluster.cc:1428] Setting key 7bc37719fd5b909612ef913b340ca4a9
I20260926 09:22:30.104969 31642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:30.105391 31642 log.cc:824] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:30.106151 31642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: No bootstrap required, opened a new log
I20260926 09:22:30.108457 31642 raft_consensus.cc:348] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } }
I20260926 09:22:30.108642 31642 raft_consensus.cc:374] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:30.108680 31642 raft_consensus.cc:729] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68798c8cfff45639b9b0ce71cdff09b, State: Initialized, Role: FOLLOWER
I20260926 09:22:30.108831 31642 consensus_queue.cc:260] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } }
I20260926 09:22:30.108915 31642 raft_consensus.cc:388] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:30.108945 31642 raft_consensus.cc:482] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:30.108990 31642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:30.109673 31642 raft_consensus.cc:504] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } }
I20260926 09:22:30.109763 31642 leader_election.cc:302] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e68798c8cfff45639b9b0ce71cdff09b; no voters:
I20260926 09:22:30.109954 31642 leader_election.cc:288] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:30.110141 31648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:30.110258 31642 sys_catalog.cc:564] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:30.110380 31648 raft_consensus.cc:686] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 LEADER]: Becoming Leader. State: Replica: e68798c8cfff45639b9b0ce71cdff09b, State: Running, Role: LEADER
I20260926 09:22:30.110530 31648 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } }
I20260926 09:22:30.110930 31648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68798c8cfff45639b9b0ce71cdff09b. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } }
I20260926 09:22:30.111009 31648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:30.111753 31656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:30.111974 31647 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } }
I20260926 09:22:30.112046 31647 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:30.112243 31656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:30.114110 31656 catalog_manager.cc:1347] Generated new cluster ID: 6bfa86a02d4941d9a2e7a94a2e69b98a
I20260926 09:22:30.114164 31656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:30.116461 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.105344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.24.235.126:43935)
0926 09:22:30.105541 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.105552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.105586 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:22:30.106149 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.106157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.106247 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.106381 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.107268 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.108206 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.109269 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.109722 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.109880 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.113035 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.113040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.113041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.113054 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.114874 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.115304 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.115307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.115307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.115405 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.115615 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.115616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.115617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.115864 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.116028 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.116136 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.116277 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260926 09:22:30.143908 31656 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:30.144685 31656 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:30.149703 31656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Generated new TSK 0
I20260926 09:22:30.149916 31656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
I20260926 09:22:30.184116 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:30.270164 31668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:30.270395 31668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:30.270423 31668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:30.271690 31668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:30.271736 31668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:30.271765 31668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:30.273428 31668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:30.273663 31668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:30.273872 31668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.279336 31674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.279311 31673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.279314 31676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.279574 31668 server_base.cc:1034] running on GCE node
I20260926 09:22:30.279757 31668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.279953 31668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.281088 31668 hybrid_clock.cc:648] HybridClock initialized: now 1790414550281070 us; error 31 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.283628 31668 init.cc:377] Logged in from keytab as oryx/127.24.235.65@KRBTEST.COM (short username oryx)
I20260926 09:22:30.284021 31668 webserver.cc:460] Webserver started at http://127.24.235.65:34447/ using document root <none> and password file <none>
I20260926 09:22:30.284250 31668 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.284294 31668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.284436 31668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:30.285576 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "3cab84c9981ee50517daa33a81419500"
server_key_iv: "103a0f70cbc004c5977f25ea1448586a"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.285938 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "3cab84c9981ee50517daa33a81419500"
server_key_iv: "103a0f70cbc004c5977f25ea1448586a"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.287343 31668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:30.288200 31683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.288393 31668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:30.288497 31668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "3cab84c9981ee50517daa33a81419500"
server_key_iv: "103a0f70cbc004c5977f25ea1448586a"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.288631 31668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.298237 31668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.300684 31668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:30.300853 31668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.301185 31668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:30.301519 31668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:30.301576 31668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.301641 31668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:30.301672 31668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.307561 31668 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:40347
I20260926 09:22:30.307610 31796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:40347 every 8 connection(s)
I20260926 09:22:30.307948 31668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:30.309748 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31668
I20260926 09:22:30.309862 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:30.310086 25517 external_mini_cluster.cc:1428] Setting key 1681aee3b234cf2f3df08910ab6bbf2a
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:22:30.319512 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.308736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:34665 (local address 127.24.235.126:43935)
0926 09:22:30.308810 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.308812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.309047 (+ 235us) server_negotiation.cc:407] Connection header received
0926 09:22:30.309581 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.309583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.309596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.309719 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.311103 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.311685 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.313005 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.313344 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.313436 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.316109 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.316111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.316111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.316124 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.317785 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.318302 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.318304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.318304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.318369 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.318653 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.318654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.318655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.318821 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.318910 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.319310 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.319412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:22:30.319741 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.308699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:34665)
0926 09:22:30.308885 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.308928 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.309477 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.309868 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.309874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.310643 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.310960 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.310967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.311779 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.311780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.312904 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.312906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.313444 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.313445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.313573 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.314173 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.314182 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.315961 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.317893 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.317903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.317906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.318168 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.318450 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.318451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.318451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.318578 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.318909 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.318917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.319146 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.319278 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.319462 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260926 09:22:30.320394 31797 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:30.320493 31797 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:30.320657 31797 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:30.321246 31606 ts_manager.cc:194] Registered new tserver with Master: 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
I20260926 09:22:30.321894 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:34665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
I20260926 09:22:30.348435 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:30.431640 31802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:30.431902 31802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:30.431943 31802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:30.433254 31802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:30.433331 31802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:30.433373 31802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:30.434923 31802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:30.435209 31802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:30.435451 31802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.440915 31807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.440912 31808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.440907 31810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.440949 31802 server_base.cc:1034] running on GCE node
I20260926 09:22:30.441474 31802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.441735 31802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.442924 31802 hybrid_clock.cc:648] HybridClock initialized: now 1790414550442912 us; error 48 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.445694 31802 init.cc:377] Logged in from keytab as oryx/127.24.235.66@KRBTEST.COM (short username oryx)
I20260926 09:22:30.446180 31802 webserver.cc:460] Webserver started at http://127.24.235.66:39131/ using document root <none> and password file <none>
I20260926 09:22:30.446422 31802 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.446489 31802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.446646 31802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:30.447695 31802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "83ca21e2ed2140ca850897b44cb77d85"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "a94586e7d4c54d289a719906c3a8bfcd"
server_key_iv: "d1ad83a604af3ff51bd740bc29e7fc71"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.448043 31802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "83ca21e2ed2140ca850897b44cb77d85"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "a94586e7d4c54d289a719906c3a8bfcd"
server_key_iv: "d1ad83a604af3ff51bd740bc29e7fc71"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.449576 31802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:30.450448 31817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.450729 31802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:30.450831 31802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "83ca21e2ed2140ca850897b44cb77d85"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "a94586e7d4c54d289a719906c3a8bfcd"
server_key_iv: "d1ad83a604af3ff51bd740bc29e7fc71"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.450907 31802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.464659 31802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.467177 31802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:30.467355 31802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.467623 31802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:30.467957 31802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:30.468014 31802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.468050 31802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:30.468081 31802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.474550 31802 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:35751
I20260926 09:22:30.474733 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:35751 every 8 connection(s)
I20260926 09:22:30.475021 31802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:30.484273 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31802
I20260926 09:22:30.484416 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:30.484647 25517 external_mini_cluster.cc:1428] Setting key 836faccdfeef6702b05bb32ce98295e7
I20260926 09:22:30.485713 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.476052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:54503 (local address 127.24.235.126:43935)
0926 09:22:30.476114 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.476116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.476394 (+ 278us) server_negotiation.cc:407] Connection header received
0926 09:22:30.477007 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.477008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.477044 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.477103 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.478132 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.478655 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.479734 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.479945 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.480032 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.482541 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.482543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.482544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.482554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.483926 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.484568 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.484569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.484569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.484634 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.484923 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.484924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.484925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.485103 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.485179 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.485508 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.485599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:22:30.485864 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.476030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:54503)
0926 09:22:30.476241 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.476300 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.476891 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.477181 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.477186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.477808 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.478015 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.478019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.478767 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.478768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.479636 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.479639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.480044 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.480045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.480183 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.480733 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.480751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.482431 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.484100 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.484108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.484110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.484443 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.484708 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.484709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.484710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.484845 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.485182 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.485188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.485347 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.485473 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.485621 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260926 09:22:30.486325 31931 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:30.486436 31931 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:30.486639 31931 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:30.487038 31606 ts_manager.cc:194] Registered new tserver with Master: 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751)
I20260926 09:22:30.487596 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:54503
WARNING: no policy specified for oryx/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
I20260926 09:22:30.519819 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:30.602964 31936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:30.603178 31936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:30.603201 31936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:30.604285 31936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:30.604321 31936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:30.604347 31936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:30.605739 31936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:30.605980 31936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:30.606210 31936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.611527 31942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.611603 31936 server_base.cc:1034] running on GCE node
W20260926 09:22:30.611536 31941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.611536 31944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.611951 31936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.612251 31936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.613406 31936 hybrid_clock.cc:648] HybridClock initialized: now 1790414550613403 us; error 36 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.616001 31936 init.cc:377] Logged in from keytab as oryx/127.24.235.67@KRBTEST.COM (short username oryx)
I20260926 09:22:30.616575 31936 webserver.cc:460] Webserver started at http://127.24.235.67:35539/ using document root <none> and password file <none>
I20260926 09:22:30.616817 31936 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.616861 31936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.616956 31936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:30.618072 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "08b113f19e954379945e43267c02b20e"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "80cb15a6497ddcaabdba752cb71886de"
server_key_iv: "c0faa3a59261a8487b1a9dc72f086876"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.618386 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "08b113f19e954379945e43267c02b20e"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "80cb15a6497ddcaabdba752cb71886de"
server_key_iv: "c0faa3a59261a8487b1a9dc72f086876"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.619738 31936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:30.620518 31951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.620747 31936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:30.620857 31936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "08b113f19e954379945e43267c02b20e"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "80cb15a6497ddcaabdba752cb71886de"
server_key_iv: "c0faa3a59261a8487b1a9dc72f086876"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.620935 31936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.633903 31936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.636279 31936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:30.636447 31936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.636698 31936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:30.637063 31936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:30.637130 31936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.637171 31936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:30.637198 31936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.643060 31936 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:43077
I20260926 09:22:30.643116 32064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:43077 every 8 connection(s)
I20260926 09:22:30.643417 31936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:30.645259 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 31936
I20260926 09:22:30.645370 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:30.645540 25517 external_mini_cluster.cc:1428] Setting key aae13f8c6357f68097905f069d32acf4
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:30.652948 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.644155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:51925 (local address 127.24.235.126:43935)
0926 09:22:30.644218 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.644221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.644483 (+ 262us) server_negotiation.cc:407] Connection header received
0926 09:22:30.644955 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.644956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.644966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.645073 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.646324 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.646895 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.647890 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.648073 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.648170 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.650474 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.650475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.650476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.650482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.651696 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.652047 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.652048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.652048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.652096 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.652346 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.652348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.652348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.652499 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.652590 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.652801 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.652871 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 09:22:30.653146 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.644186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:51925)
0926 09:22:30.644360 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.644396 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.644871 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.645187 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.645190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.645943 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.646227 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.646233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.646971 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.646972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.647801 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.647803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.648161 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.648162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.648289 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.648872 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.648880 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.650395 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.651776 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.651781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.651783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.651983 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.652153 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.652154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.652154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.652261 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.652575 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.652579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.652697 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.652775 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.652889 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 09:22:30.653645 32065 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:30.653769 32065 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:30.653944 32065 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:30.654296 31606 ts_manager.cc:194] Registered new tserver with Master: 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:30.654690 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:51925
I20260926 09:22:30.657334 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.125.keytab.
Entry for principal oryx/127.24.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.125.keytab.
I20260926 09:22:30.685089 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:38855
--webserver_interface=127.24.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.125.keytab
--principal=oryx/127.24.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43935,127.24.235.125:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:30.775194 32070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:30.775413 32070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:30.775436 32070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:30.776742 32070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:30.776789 32070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:30.776804 32070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:30.776818 32070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:30.776829 32070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:30.778286 32070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43935,127.24.235.125:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.125.keytab
--principal=oryx/127.24.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
W20260926 09:22:30.778391 32070 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.235.126:43935,127.24.235.125:38855'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 09:22:30.778506 32070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:30.778719 32070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.784507 32076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.784519 32075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.784574 32070 server_base.cc:1034] running on GCE node
W20260926 09:22:30.784583 32078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.785110 32070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.785367 32070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.786525 32070 hybrid_clock.cc:648] HybridClock initialized: now 1790414550786474 us; error 64 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.789605 32070 init.cc:377] Logged in from keytab as oryx/127.24.235.125@KRBTEST.COM (short username oryx)
I20260926 09:22:30.790119 32070 webserver.cc:460] Webserver started at http://127.24.235.125:33071/ using document root <none> and password file <none>
I20260926 09:22:30.790357 32070 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.790426 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.790580 32070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:30.791818 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/instance:
uuid: "10743bbd99de4fb7bcbf99d6336c5024"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "ea9d1ee7528adb0cc45e1e12f0b26ba5"
server_key_iv: "dfb59ba096bc5f2557e6f42c525a4aa5"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.792203 32070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal/instance:
uuid: "10743bbd99de4fb7bcbf99d6336c5024"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "ea9d1ee7528adb0cc45e1e12f0b26ba5"
server_key_iv: "dfb59ba096bc5f2557e6f42c525a4aa5"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.793771 32070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:30.794677 32085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.794883 32070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:30.794993 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "10743bbd99de4fb7bcbf99d6336c5024"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "ea9d1ee7528adb0cc45e1e12f0b26ba5"
server_key_iv: "dfb59ba096bc5f2557e6f42c525a4aa5"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.795073 32070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.811014 32070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.813629 32070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:30.813812 32070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.817982 32070 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.125:38855
I20260926 09:22:30.818019 32137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.125:38855 every 8 connection(s)
I20260926 09:22:30.818375 32070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
I20260926 09:22:30.819415 32138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:30.820350 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 32070
I20260926 09:22:30.821573 32138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.125@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:30.831333 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.822676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.24.235.126:43935)
0926 09:22:30.822744 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.822747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.822944 (+ 197us) server_negotiation.cc:407] Connection header received
0926 09:22:30.823040 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.823041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.823052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.823149 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.823869 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.824671 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.825741 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.826010 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.826087 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.828578 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.828580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.828581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.828589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.830012 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830410 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.830411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.830412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.830472 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830712 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.830713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.830713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.830868 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.830949 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.831145 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.831231 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:22:30.831553 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.822542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41090)
0926 09:22:30.822829 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.822866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.822965 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.823174 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.823177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.823576 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.823765 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.823770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.824822 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.824824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.825646 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.825649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.826105 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.826106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.826168 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.826856 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.826865 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.828483 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.830107 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.830115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.830323 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.830541 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.830542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.830640 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.830939 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.830945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.831014 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.831122 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.831301 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260926 09:22:30.831566 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.821139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.24.235.125:38855)
0926 09:22:30.821305 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.821315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.821348 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:22:30.821924 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.821934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.822002 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.822158 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.823017 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.823955 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.824751 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.825239 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.825369 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.828062 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.828070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.828070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.828077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.830048 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830451 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.830454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.830454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.830542 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.830761 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.830762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.830762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.830978 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.831130 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.831248 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.831375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260926 09:22:30.832216 32138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.125@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
I20260926 09:22:30.838413 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.832479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34694)
0926 09:22:30.832528 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.832540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.832606 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.832707 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.832708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.832814 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.832860 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.832861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.833497 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.833498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.833971 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.833973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.834359 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.834359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.834412 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.834829 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.834833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.835974 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.837595 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.837596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.837596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.837751 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.837921 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.837921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.837922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.837970 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.838181 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.838181 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.838215 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.838267 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.838344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:22:30.838531 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.832505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.24.235.125:38855)
0926 09:22:30.832531 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.832534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.832579 (+ 45us) server_negotiation.cc:407] Connection header received
0926 09:22:30.832658 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.832659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.832667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.832703 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.832907 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.833432 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.834068 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.834265 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.834339 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.836049 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.836050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.836050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.836056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.837517 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.837815 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.837816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.837816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.837864 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.838011 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.838011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.838011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.838100 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.838143 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.838302 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.838390 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":14}
I20260926 09:22:30.839856 32138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Bootstrap starting.
I20260926 09:22:30.840633 32138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:30.840873 32138 log.cc:824] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:30.841694 32138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: No bootstrap required, opened a new log
I20260926 09:22:30.843123 32138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } }
I20260926 09:22:30.843253 32138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:30.843276 32138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10743bbd99de4fb7bcbf99d6336c5024, State: Initialized, Role: FOLLOWER
I20260926 09:22:30.843412 32138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } }
I20260926 09:22:30.843683 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } } }
I20260926 09:22:30.843757 32138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:30.843767 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:22:30.845644 32155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:22:30.845752 32155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:30.847424 32070 master_runner.cc:428] Detected that this master 10743bbd99de4fb7bcbf99d6336c5024 is joining an existing cluster
I20260926 09:22:30.847484 32070 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.235.125:38855
I20260926 09:22:30.855399 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.848616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.24.235.126:43935)
0926 09:22:30.848662 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.848664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.848959 (+ 295us) server_negotiation.cc:407] Connection header received
0926 09:22:30.849064 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.849065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.849076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.849138 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.849528 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.850303 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851081 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.851292 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851367 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.852457 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.852458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.852459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.852466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.854207 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854599 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.854600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.854601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.854658 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854867 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.854869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.854869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.854987 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.855058 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.855228 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.855303 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:22:30.855414 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.848524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41106)
0926 09:22:30.848870 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.848881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.848960 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.849143 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.849144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.849365 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.849444 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.849447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.850392 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.850392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.850992 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.850993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.851379 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.851449 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.851941 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.851946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.852367 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.854300 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.854302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.854514 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.854737 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.854738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.854796 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.855060 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.855062 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.855104 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.855202 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.855303 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260926 09:22:30.855557 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.848815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.24.235.125:38855)
0926 09:22:30.848851 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.848852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.848978 (+ 126us) server_negotiation.cc:407] Connection header received
0926 09:22:30.849097 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.849104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.849113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.849160 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.849570 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.850297 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851414 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.851688 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851726 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.852708 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.852709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.852709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.852715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.854491 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854852 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.854853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.854853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.854900 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.855074 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.855075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.855075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.855193 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.855232 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.855399 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.855467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:30.855751 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.848788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34698)
0926 09:22:30.848919 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.848930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.849001 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.849197 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.849197 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.849413 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.849481 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.849483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.850386 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.850387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.851320 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.851323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.851795 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.851795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.851848 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.852194 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.852198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.852616 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.854575 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.854576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.854776 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.854963 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.854964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.854964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.855013 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.855271 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.855272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.855304 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.855589 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.855662 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.126@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
I20260926 09:22:30.866798 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.858729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.24.235.125:38855)
0926 09:22:30.858767 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.858769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.858951 (+ 182us) server_negotiation.cc:407] Connection header received
0926 09:22:30.859022 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.859022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.859031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.859072 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.859557 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.859994 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.861134 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.861329 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.861383 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.863546 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.863548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.863548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.863566 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.865367 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.865875 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.865876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.865877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.865944 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.866191 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.866192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.866192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.866316 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.866400 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.866602 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.866692 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:30.866815 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.858659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34700)
0926 09:22:30.858826 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.858867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.858949 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.859088 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.859091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.859425 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.859486 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.859488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.860071 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.860072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.861006 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.861008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.861403 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.861404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.861506 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.861948 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.861953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.863440 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.865509 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.865518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.865520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.865765 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.866035 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.866036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.866036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.866111 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.866391 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.866399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.866462 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.866575 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.866710 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260926 09:22:30.867652 31606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.235.125:38855: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024"
member_type: NON_VOTER
last_known_addr {
host: "127.24.235.125"
port: 38855
}
attrs {
promote: true
}
}
dest_uuid: "e68798c8cfff45639b9b0ce71cdff09b"
cas_config_opid_index: -1
I20260926 09:22:30.867998 31606 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } }
I20260926 09:22:30.868506 32165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } }
I20260926 09:22:30.868835 32112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:30.869542 32165 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.235.125:38855
I20260926 09:22:30.869764 32070 master.cc:537] Master@127.24.235.125:38855 shutting down...
I20260926 09:22:30.869824 32112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Refusing update from remote peer e68798c8cfff45639b9b0ce71cdff09b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 09:22:30.869803 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } }
I20260926 09:22:30.869884 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:30.870532 32164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:22:30.871348 31591 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 09:22:30.871810 31591 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b -> Peer 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125:38855): Couldn't send request to peer 10743bbd99de4fb7bcbf99d6336c5024. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:22:30.872849 32070 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:30.872951 32070 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:30.873003 32070 tablet_replica.cc:330] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: stopping tablet replica
I20260926 09:22:30.876029 32070 master.cc:552] Master@127.24.235.125:38855 shutdown complete.
I20260926 09:22:30.876082 32070 master_runner.cc:305] Clearing existing system tablet
I20260926 09:22:30.876221 32070 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.876269 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.877095 32070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:22:30.877734 32169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.877928 32070 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.877993 32070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:30.878021 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "10743bbd99de4fb7bcbf99d6336c5024"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "ea9d1ee7528adb0cc45e1e12f0b26ba5"
server_key_iv: "dfb59ba096bc5f2557e6f42c525a4aa5"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.878803 32070 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:30.880471 32070 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:22:30.880563 32070 log.cc:1196] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 09:22:30.880769 32070 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Deleting consensus metadata
I20260926 09:22:30.880911 32070 master_runner.cc:315] Copying system tablet from 127.24.235.126:43935
I20260926 09:22:30.881626 32070 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: tablet copy: Beginning tablet copy session from remote peer at address 127.24.235.126:43935
I20260926 09:22:30.888145 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.882109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.24.235.126:43935)
0926 09:22:30.882170 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.882173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.882310 (+ 137us) server_negotiation.cc:407] Connection header received
0926 09:22:30.882396 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.882397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.882408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.882467 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.882853 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.883460 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.884220 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.884432 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.884523 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.885570 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.885571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.885571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.885578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.886961 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.887320 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.887322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.887322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.887380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.887595 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.887596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.887596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.887722 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.887793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.887967 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.888047 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:22:30.888388 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.882036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41112)
0926 09:22:30.882230 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:30.882240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:30.882328 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:30.882473 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:30.882474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:30.882681 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:30.882730 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.882733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.883548 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.883549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.884129 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:30.884131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.884525 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.884526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:30.884593 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.885139 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:30.885144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:30.885454 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:30.887053 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.887054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.887055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.887242 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.887458 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:30.887459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:30.887459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:30.887518 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:30.887795 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:30.887797 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:30.887843 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 09:22:30.887938 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:22:30.888049 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260926 09:22:30.888818 31626 tablet_copy_service.cc:140] P e68798c8cfff45639b9b0ce71cdff09b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 10743bbd99de4fb7bcbf99d6336c5024 ({username='oryx', principal='oryx/127.24.235.125@KRBTEST.COM'} at 127.0.0.1:41112)
I20260926 09:22:30.888937 31626 tablet_copy_service.cc:161] P e68798c8cfff45639b9b0ce71cdff09b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 10743bbd99de4fb7bcbf99d6336c5024 at {username='oryx', principal='oryx/127.24.235.125@KRBTEST.COM'} at 127.0.0.1:41112: session id = 10743bbd99de4fb7bcbf99d6336c5024-00000000000000000000000000000000
I20260926 09:22:30.889761 31626 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:22:30.890436 32070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:22:30.891608 32070 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: tablet copy: Starting download of 0 data blocks...
I20260926 09:22:30.891765 32070 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: tablet copy: Starting download of 1 WAL segments...
I20260926 09:22:30.892815 32070 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:22:30.893690 31626 tablet_copy_service.cc:342] P e68798c8cfff45639b9b0ce71cdff09b: Request end of tablet copy session 10743bbd99de4fb7bcbf99d6336c5024-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.235.125@KRBTEST.COM'} at 127.0.0.1:41112
I20260926 09:22:30.893754 31626 tablet_copy_service.cc:433] P e68798c8cfff45639b9b0ce71cdff09b: ending tablet copy session 10743bbd99de4fb7bcbf99d6336c5024-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 10743bbd99de4fb7bcbf99d6336c5024
I20260926 09:22:30.896049 32070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:30.897308 32179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:30.897413 32176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.897538 32070 server_base.cc:1034] running on GCE node
W20260926 09:22:30.897647 32177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:30.897809 32070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:30.897964 32070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:30.899091 32070 hybrid_clock.cc:648] HybridClock initialized: now 1790414550899083 us; error 26 us; skew 500 ppm
Sep 26 09:22:30 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:30.900511 32070 init.cc:377] Logged in from keytab as oryx/127.24.235.125@KRBTEST.COM (short username oryx)
I20260926 09:22:30.900784 32070 webserver.cc:460] Webserver started at http://127.24.235.125:40395/ using document root <none> and password file <none>
I20260926 09:22:30.900921 32070 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:30.900969 32070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:30.901659 32070 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.902302 32186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:30.902439 32070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:30.902509 32070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "10743bbd99de4fb7bcbf99d6336c5024"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "ea9d1ee7528adb0cc45e1e12f0b26ba5"
server_key_iv: "dfb59ba096bc5f2557e6f42c525a4aa5"
server_key_version: "encryptionkey@0"
I20260926 09:22:30.902642 32070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:30.912724 32070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:30.914245 32070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:30.917270 32070 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.125:38855
I20260926 09:22:30.917539 32238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.125:38855 every 8 connection(s)
I20260926 09:22:30.917675 32070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
I20260926 09:22:30.918573 32239 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:22:30.919031 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Bootstrap starting.
I20260926 09:22:30.921957 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:22:30.922199 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: Bootstrap complete.
I20260926 09:22:30.922410 32239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } }
I20260926 09:22:30.922502 32239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10743bbd99de4fb7bcbf99d6336c5024, State: Initialized, Role: LEARNER
I20260926 09:22:30.922544 32239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } }
I20260926 09:22:30.922719 32239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:30.922776 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } }
I20260926 09:22:30.922914 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:22:30.924256 32253 catalog_manager.cc:1259] Loaded cluster ID: 6bfa86a02d4941d9a2e7a94a2e69b98a
I20260926 09:22:30.924311 32253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: loading cluster ID for follower catalog manager: success
I20260926 09:22:30.925436 32253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: acquiring CA information for follower catalog manager: success
I20260926 09:22:30.925732 32253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:22:30.941074 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:30.933553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.24.235.125:38855)
0926 09:22:30.933703 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:30.933705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:30.933720 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:22:30.933744 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:30.933745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:30.933778 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:30.933841 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:30.934657 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.935350 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.936347 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:30.936569 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:30.936631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:30.938261 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:30.938263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:30.938263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:30.938270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:30.939813 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.940236 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.940237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.940237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.940290 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:30.940516 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:30.940517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:30.940517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:30.940631 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:30.940706 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:30.940882 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 09:22:30.940946 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.24.235.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.124.keytab.
Entry for principal oryx/127.24.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.124.keytab.
I20260926 09:22:30.968155 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:33641
--webserver_interface=127.24.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.124.keytab
--principal=oryx/127.24.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:31.059116 32257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:31.059360 32257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:31.059386 32257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:31.060885 32257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:31.060933 32257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:31.060949 32257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:31.060964 32257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:31.060977 32257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:31.062740 32257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.124.keytab
--principal=oryx/127.24.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:31.063027 32257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:31.063292 32257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:31.069006 32265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.069155 32257 server_base.cc:1034] running on GCE node
W20260926 09:22:31.069067 32262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.069096 32263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.069469 32257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:31.069832 32257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:31.071007 32257 hybrid_clock.cc:648] HybridClock initialized: now 1790414551070991 us; error 25 us; skew 500 ppm
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:31.073544 32257 init.cc:377] Logged in from keytab as oryx/127.24.235.124@KRBTEST.COM (short username oryx)
I20260926 09:22:31.074002 32257 webserver.cc:460] Webserver started at http://127.24.235.124:40715/ using document root <none> and password file <none>
I20260926 09:22:31.074245 32257 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.074314 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.074456 32257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:31.075536 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/instance:
uuid: "99213f8260384145a89fd03e3266fe94"
format_stamp: "Formatted at 2026-09-26 09:22:31 on dist-test-slave-7zlw"
server_key: "e64b0fef37a05ee020d7ee2ae35be663"
server_key_iv: "5808c6b083e51e341e8b3b028f422ae3"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.075938 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal/instance:
uuid: "99213f8260384145a89fd03e3266fe94"
format_stamp: "Formatted at 2026-09-26 09:22:31 on dist-test-slave-7zlw"
server_key: "e64b0fef37a05ee020d7ee2ae35be663"
server_key_iv: "5808c6b083e51e341e8b3b028f422ae3"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.077365 32257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:31.078207 32272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.078500 32257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:31.078644 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "99213f8260384145a89fd03e3266fe94"
format_stamp: "Formatted at 2026-09-26 09:22:31 on dist-test-slave-7zlw"
server_key: "e64b0fef37a05ee020d7ee2ae35be663"
server_key_iv: "5808c6b083e51e341e8b3b028f422ae3"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.078723 32257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.087962 32257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:31.089824 32257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:31.089986 32257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:31.097704 32257 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.124:33641
I20260926 09:22:31.097836 32324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.124:33641 every 8 connection(s)
I20260926 09:22:31.098227 32257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
I20260926 09:22:31.099596 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:31.101886 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:22:31.103912 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 32257
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.124@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
I20260926 09:22:31.117821 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.102782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.24.235.126:43935)
0926 09:22:31.102845 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.102847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.103196 (+ 349us) server_negotiation.cc:407] Connection header received
0926 09:22:31.103793 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.103794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.103807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.103868 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.104728 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.105303 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.106305 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.106496 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.106580 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.114363 (+ 7783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.114364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.114365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.114381 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.116111 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.116591 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.116593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.116593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.116678 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.117054 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.117055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.117055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.117211 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.117267 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.117639 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.117724 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:22:31.118162 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.102733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41122)
0926 09:22:31.103066 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.103102 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.103595 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.103945 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.103948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.104339 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.104633 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.104644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.105416 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.105417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.106198 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.106200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.106611 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.106612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.106741 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.112677 (+ 5936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.112688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.114262 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.116282 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.116288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.116290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.116508 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.116768 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.116768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.116768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.116894 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.117281 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.117288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.117474 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.117640 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.117777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":17,"thread_start_us":171,"threads_started":1}
I20260926 09:22:31.118623 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.104430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.24.235.124:33641)
0926 09:22:31.104579 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.104584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.104611 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:22:31.104634 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.104636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.104650 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.104739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.105415 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.106225 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.106924 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.107227 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.107299 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.114903 (+ 7604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.114910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.114910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.114916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.116889 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.117350 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.117352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.117353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.117400 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.117625 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.117625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.117626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.117823 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.118154 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.118205 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.118448 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":122,"thread_start_us":100,"threads_started":1}
I20260926 09:22:31.118916 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.124@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
I20260926 09:22:31.125487 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.119196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34732)
0926 09:22:31.119260 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.119271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.119319 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.119461 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.119462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.119618 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.119657 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.119659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.120312 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.120313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.120815 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.120817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.121260 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.121261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.121319 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.121769 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.121774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.123006 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.124635 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.124635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.124636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.124788 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.124943 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.124944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.124944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.124984 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.125254 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.125255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.125293 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.125354 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.125424 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:22:31.125522 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.119290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.24.235.125:38855)
0926 09:22:31.119336 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.119338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.119347 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:31.119374 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.119375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.119387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.119460 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.119719 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.120216 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.120897 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.121187 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.121264 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.123081 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.123083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.123083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.123089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.124554 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.124852 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.124853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.124853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.124902 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.125079 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.125080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.125081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.125197 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.125250 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.125384 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.125453 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:31.125972 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.124@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
I20260926 09:22:31.132090 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.126254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50758)
0926 09:22:31.126280 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.126289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.126320 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.126422 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.126423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.126526 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.126555 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.126555 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.127394 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.127395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.127870 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.127871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.128351 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.128351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.128387 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.128825 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.128829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.129994 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.131337 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.131338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.131339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.131489 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.131648 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.131649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.131649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.131695 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.131861 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.131861 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.131893 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.131941 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.132011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12}
I20260926 09:22:31.132156 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.126279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.24.235.124:33641)
0926 09:22:31.126330 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.126332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.126353 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:22:31.126376 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.126377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.126384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.126420 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.126607 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.127238 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.127958 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.128279 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.128345 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.130078 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.130079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.130080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.130086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.131258 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.131549 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.131550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.131550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.131595 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.131731 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.131732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.131732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.131817 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.131859 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.131954 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.132054 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":32}
I20260926 09:22:31.134263 32325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Bootstrap starting.
I20260926 09:22:31.135690 32325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:31.136404 32325 log.cc:824] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:31.137262 32325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: No bootstrap required, opened a new log
I20260926 09:22:31.140228 32325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } }
I20260926 09:22:31.140390 32325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:31.140437 32325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99213f8260384145a89fd03e3266fe94, State: Initialized, Role: FOLLOWER
I20260926 09:22:31.140605 32325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } }
I20260926 09:22:31.141057 32325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:31.141083 32330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } } }
I20260926 09:22:31.141216 32330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:22:31.143486 32341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:22:31.143555 32341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:31.146008 32257 master_runner.cc:428] Detected that this master 99213f8260384145a89fd03e3266fe94 is joining an existing cluster
I20260926 09:22:31.146106 32257 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.235.124:33641
I20260926 09:22:31.155138 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.24.235.125:38855)
0926 09:22:31.147688 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.147690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.147896 (+ 206us) server_negotiation.cc:407] Connection header received
0926 09:22:31.147989 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.147990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.148001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.148032 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.149269 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150011 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.151046 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.151318 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.151384 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.152297 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.152299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.152300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.152308 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.153959 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.154323 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.154325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.154325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.154382 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.154605 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.154606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.154606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.154751 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.154814 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.154961 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.155040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 09:22:31.155366 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34734)
0926 09:22:31.147834 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.147845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.147918 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.148095 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.148095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.149136 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.149189 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.149191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150101 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.150948 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.150950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.151436 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.151437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.151509 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.151891 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.151895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.152206 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.154052 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.154053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.154054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.154247 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.154462 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.154462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.154463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.154534 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.154805 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.154805 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.154848 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.154938 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.155042 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":79,"threads_started":1}
I20260926 09:22:31.170715 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31668
I20260926 09:22:31.177664 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.24.235.124:33641)
0926 09:22:31.147794 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.147796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.147896 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:22:31.147986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.147986 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.147994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.148024 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.148560 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.149334 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150379 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150668 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150703 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.174171 (+ 23468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.174173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.174173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.174179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.176488 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.176915 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.176916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.176916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.176958 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.177162 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.177162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.177163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.177289 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.177327 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.177489 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.177566 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:31.178021 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50774)
0926 09:22:31.147844 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.147852 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.147918 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.148078 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.148079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.148415 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.148475 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.148477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.149417 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.149418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.150300 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.150302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150781 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.150832 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.151355 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.151362 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.174046 (+ 22684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.176636 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.176637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.176638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.176838 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.177040 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.177041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.177041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.177089 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.177364 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.177364 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.177397 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.177752 (+ 355us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.177849 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":33,"thread_start_us":58,"threads_started":1}
I20260926 09:22:31.178874 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.24.235.126:43935)
0926 09:22:31.147480 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.147482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.147756 (+ 274us) server_negotiation.cc:407] Connection header received
0926 09:22:31.147865 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.147866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.147880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.147919 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.148431 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.148989 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150455 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150752 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150798 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.174850 (+ 24052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.174852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.174852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.174862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.176733 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.177280 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.177281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.177281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.177352 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.177622 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.177623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.177623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.177797 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.177853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.178628 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.178724 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:31.179179 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:40347
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:43935
--webserver_port=34447
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:22:31.179270 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.147326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41136)
0926 09:22:31.147687 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.147699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.147768 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.147986 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.147987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.148279 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.148360 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.148362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.149433 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.149434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.150350 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.150353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.150874 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.150875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.150945 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.173864 (+ 22919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.173873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.174708 (+ 835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.176872 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.176873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.176874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.177152 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.177445 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.177446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.177447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.177518 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.178422 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.178423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.178479 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.179005 (+ 526us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.179134 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414550, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.126@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
I20260926 09:22:31.189229 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.181564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50790)
0926 09:22:31.181647 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.181659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.181700 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.181885 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.181886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.182085 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.182164 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.182166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.183121 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.183122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.183711 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.183714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.184250 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.184251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.184322 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.184826 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.184832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.186267 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.188006 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.188007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.188008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.188215 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.188450 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.188451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.188452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.188514 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.188811 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.188812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.188860 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.188955 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.189112 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 09:22:31.189596 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.181656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.24.235.124:33641)
0926 09:22:31.181704 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.181705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.181715 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:31.181794 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.181795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.181805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.181865 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.182248 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.183029 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.183925 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.184153 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.184248 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.186376 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.186377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.186378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.186385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.187893 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.188311 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.188312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.188313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.188375 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.188600 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.188601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.188601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.188726 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.188805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.188982 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.189221 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:22:31.190160 31606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.235.124:33641: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "99213f8260384145a89fd03e3266fe94"
member_type: NON_VOTER
last_known_addr {
host: "127.24.235.124"
port: 33641
}
attrs {
promote: true
}
}
dest_uuid: "e68798c8cfff45639b9b0ce71cdff09b"
cas_config_opid_index: 5
I20260926 09:22:31.190466 31606 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } }
I20260926 09:22:31.190847 32164 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 99213f8260384145a89fd03e3266fe94 (127.24.235.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.191758 32164 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.235.124:33641
I20260926 09:22:31.191845 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.191921 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.192297 32257 master.cc:537] Master@127.24.235.124:33641 shutting down...
W20260926 09:22:31.193408 31591 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b -> Peer 99213f8260384145a89fd03e3266fe94 (127.24.235.124:33641): Couldn't send request to peer 99213f8260384145a89fd03e3266fe94. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 09:22:31.195775 32257 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:31.195878 32257 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:31.195926 32257 tablet_replica.cc:330] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: stopping tablet replica
I20260926 09:22:31.198307 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.192309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.24.235.125:38855)
0926 09:22:31.192356 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.192358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.193562 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 09:22:31.193681 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.193682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.193708 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.193752 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.194345 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.195198 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.196800 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.197526 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.197590 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.197671 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.198159 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.198211 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:31.198764 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.192244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:34748)
0926 09:22:31.193487 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.193499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.193596 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.193926 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.193927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.193929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.194240 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.194243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.195340 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.195340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.196682 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.196686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.197951 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.197952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.198032 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.198058 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.198625 (+ 567us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.198685 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227}
I20260926 09:22:31.199220 32213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 LEARNER]: Refusing update from remote peer e68798c8cfff45639b9b0ce71cdff09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 09:22:31.199455 32165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 09:22:31.200135 32213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 99213f8260384145a89fd03e3266fe94 (127.24.235.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.201563 32165 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: attempting to promote NON_VOTER 10743bbd99de4fb7bcbf99d6336c5024 to VOTER
I20260926 09:22:31.201628 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68798c8cfff45639b9b0ce71cdff09b. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.201712 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:22:31.201751 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: NON_VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: true } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.201792 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:22:31.201802 32165 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } }
I20260926 09:22:31.202409 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.202253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50806)
0926 09:22:31.202332 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.202374 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 09:22:31.202625 32213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 LEARNER]: Refusing update from remote peer e68798c8cfff45639b9b0ce71cdff09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 09:22:31.202958 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.202741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50816)
0926 09:22:31.202897 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.202939 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 09:22:31.202880 32353 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 09:22:31.203146 31591 consensus_peers.cc:487] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b -> Peer 99213f8260384145a89fd03e3266fe94 (127.24.235.124:33641): Couldn't send request to peer 99213f8260384145a89fd03e3266fe94. Status: Network error: Client connection negotiation failed: client connection to 127.24.235.124:33641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:22:31.203768 32164 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.204589 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.204669 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.204742 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.204795 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.205058 32213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.205892 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.206010 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:22:31.209681 32257 master.cc:552] Master@127.24.235.124:33641 shutdown complete.
I20260926 09:22:31.209757 32257 master_runner.cc:305] Clearing existing system tablet
I20260926 09:22:31.209946 32257 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.210000 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.210794 32257 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:31.211426 32355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.211599 32257 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.211663 32257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:31.211695 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "99213f8260384145a89fd03e3266fe94"
format_stamp: "Formatted at 2026-09-26 09:22:31 on dist-test-slave-7zlw"
server_key: "e64b0fef37a05ee020d7ee2ae35be663"
server_key_iv: "5808c6b083e51e341e8b3b028f422ae3"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.212497 32257 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:31.214174 32257 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 09:22:31.214279 32257 log.cc:1196] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 09:22:31.214491 32257 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Deleting consensus metadata
I20260926 09:22:31.214702 32257 master_runner.cc:315] Copying system tablet from 127.24.235.126:43935
I20260926 09:22:31.215425 32257 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: tablet copy: Beginning tablet copy session from remote peer at address 127.24.235.126:43935
I20260926 09:22:31.222513 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.215768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:41140)
0926 09:22:31.215908 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.215920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.216016 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.216190 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.216191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.216382 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.216448 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.216450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.217287 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.217288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.217922 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.217924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.218365 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.218366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.218449 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.218989 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.218995 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.219342 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.221328 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.221330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.221330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.221532 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.221762 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.221763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.221764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.221828 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.222118 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.222119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.222171 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.222271 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.222394 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260926 09:22:31.222496 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.215813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.24.235.126:43935)
0926 09:22:31.215861 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.215864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.215999 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:22:31.216106 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.216108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.216119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.216187 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.216557 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.217202 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.218036 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.218267 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.218364 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.219463 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.219465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.219465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.219473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.221220 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.221615 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.221617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.221617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.221689 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.221909 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.221910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.221910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.222043 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.222128 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.222298 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.222387 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:22:31.223193 31626 tablet_copy_service.cc:140] P e68798c8cfff45639b9b0ce71cdff09b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 99213f8260384145a89fd03e3266fe94 ({username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:41140)
I20260926 09:22:31.223292 31626 tablet_copy_service.cc:161] P e68798c8cfff45639b9b0ce71cdff09b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 99213f8260384145a89fd03e3266fe94 at {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:41140: session id = 99213f8260384145a89fd03e3266fe94-00000000000000000000000000000000
I20260926 09:22:31.223899 31626 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 09:22:31.224421 32257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 09:22:31.225781 32257 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: tablet copy: Starting download of 0 data blocks...
I20260926 09:22:31.225960 32257 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: tablet copy: Starting download of 1 WAL segments...
I20260926 09:22:31.226763 32257 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 09:22:31.227731 31626 tablet_copy_service.cc:342] P e68798c8cfff45639b9b0ce71cdff09b: Request end of tablet copy session 99213f8260384145a89fd03e3266fe94-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:41140
I20260926 09:22:31.227819 31626 tablet_copy_service.cc:433] P e68798c8cfff45639b9b0ce71cdff09b: ending tablet copy session 99213f8260384145a89fd03e3266fe94-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 99213f8260384145a89fd03e3266fe94
I20260926 09:22:31.230487 32257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:31.231834 32363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.231849 32362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.234205 32257 server_base.cc:1034] running on GCE node
W20260926 09:22:31.234272 32365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.234416 32257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:31.234572 32257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:31.235705 32257 hybrid_clock.cc:648] HybridClock initialized: now 1790414551235695 us; error 26 us; skew 500 ppm
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:31.237169 32257 init.cc:377] Logged in from keytab as oryx/127.24.235.124@KRBTEST.COM (short username oryx)
I20260926 09:22:31.237468 32257 webserver.cc:460] Webserver started at http://127.24.235.124:45231/ using document root <none> and password file <none>
I20260926 09:22:31.237605 32257 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.237637 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.238274 32257 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:31.238832 32372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.238979 32257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:31.239025 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "99213f8260384145a89fd03e3266fe94"
format_stamp: "Formatted at 2026-09-26 09:22:31 on dist-test-slave-7zlw"
server_key: "e64b0fef37a05ee020d7ee2ae35be663"
server_key_iv: "5808c6b083e51e341e8b3b028f422ae3"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.239089 32257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.258659 32257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:31.260447 32257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:31.263785 32257 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.124:33641
I20260926 09:22:31.263957 32424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.124:33641 every 8 connection(s)
I20260926 09:22:31.264155 32257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
I20260926 09:22:31.265107 32425 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:22:31.265581 32425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Bootstrap starting.
I20260926 09:22:31.267903 32425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:22:31.268153 32425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Bootstrap complete.
I20260926 09:22:31.268369 32425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } }
I20260926 09:22:31.268460 32425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99213f8260384145a89fd03e3266fe94, State: Initialized, Role: LEARNER
I20260926 09:22:31.268504 32425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } }
I20260926 09:22:31.268746 32425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:31.268735 32428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.269151 32428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:22:31.270581 32439 catalog_manager.cc:1259] Loaded cluster ID: 6bfa86a02d4941d9a2e7a94a2e69b98a
I20260926 09:22:31.270792 32439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: loading cluster ID for follower catalog manager: success
I20260926 09:22:31.271836 32439 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: acquiring CA information for follower catalog manager: success
I20260926 09:22:31.272128 32439 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 09:22:31.304338 32350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:31.304636 32350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:31.304679 32350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:31.306597 32350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:31.306675 32350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:31.306718 32350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:31.309002 32350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:40347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=34447
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:31.309336 32350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:31.309638 32350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:31.315228 32447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.315237 32444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.315237 32445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.315966 32350 server_base.cc:1034] running on GCE node
I20260926 09:22:31.316221 32350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:31.316466 32350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:31.317689 32350 hybrid_clock.cc:648] HybridClock initialized: now 1790414551317674 us; error 28 us; skew 500 ppm
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:31.320235 32350 init.cc:377] Logged in from keytab as oryx/127.24.235.65@KRBTEST.COM (short username oryx)
I20260926 09:22:31.320724 32350 webserver.cc:460] Webserver started at http://127.24.235.65:34447/ using document root <none> and password file <none>
I20260926 09:22:31.320981 32350 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.321095 32350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.322741 32350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:31.323522 32454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.323817 32350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:31.323951 32350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "3cab84c9981ee50517daa33a81419500"
server_key_iv: "103a0f70cbc004c5977f25ea1448586a"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.324405 32350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.338454 32350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:31.341344 32350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:31.341554 32350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:31.341954 32350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:31.342404 32350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:31.342442 32350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.342465 32350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:31.342482 32350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.350839 32350 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:40347
I20260926 09:22:31.350883 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:40347 every 8 connection(s)
I20260926 09:22:31.351259 32350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:31.357626 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 32350
I20260926 09:22:31.357787 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31802
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:31.363607 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:39341 (local address 127.24.235.125:38855)
0926 09:22:31.352589 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.352591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.352764 (+ 173us) server_negotiation.cc:407] Connection header received
0926 09:22:31.353196 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.353198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.353210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.353245 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.354296 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.354983 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356056 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356346 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356411 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.359980 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.359982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.359982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.359991 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.361851 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.362382 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.362383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.362383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.362439 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.362727 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.362728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.362728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.362891 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.362935 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.363365 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.363458 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:31.364459 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:56513 (local address 127.24.235.126:43935)
0926 09:22:31.352102 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.352104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.352340 (+ 236us) server_negotiation.cc:407] Connection header received
0926 09:22:31.353197 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.353198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.353210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.353271 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.354014 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.354763 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356199 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356485 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356551 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.361113 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.361115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.361115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.361123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.363267 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363647 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.363648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.363648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.363699 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363901 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.363902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.363903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.364023 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.364064 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.364287 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.364369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:22:31.364477 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:46057 (local address 127.24.235.124:33641)
0926 09:22:31.352814 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.352815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.352825 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:31.353197 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.353198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.353211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.353328 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.354394 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.355227 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356217 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356464 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356845 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.360733 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.360735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.360735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.360744 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.362901 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363319 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.363320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.363320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.363397 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363591 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.363592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.363593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.363724 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.363784 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.364288 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.364371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":82,"threads_started":1}
I20260926 09:22:31.364768 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:56513)
0926 09:22:31.352201 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.352245 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.352994 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.353366 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.353370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.353710 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.353903 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.353907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.354939 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.354940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.356103 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.356106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356561 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.356624 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.357331 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.357342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.360979 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.363352 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.363353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.363555 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.363775 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.363776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.363828 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.364112 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.364112 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.364170 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.364587 (+ 417us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.364692 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":33,"thread_start_us":70,"threads_started":1}
I20260926 09:22:31.365173 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.24.235.65:46057)
0926 09:22:31.352621 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.352631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.352995 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.353367 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.353372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.354051 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.354285 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.354287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.355328 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.355329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.356124 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.356126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356548 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.356599 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.357462 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.357466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.360615 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.363002 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.363003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.363232 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.363468 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.363469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.363470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.363524 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.363808 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.363809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.363849 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.363927 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.364147 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":64,"threads_started":1}
I20260926 09:22:31.365399 32569 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:31.365466 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.352488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.24.235.65:39341)
0926 09:22:31.352706 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.352713 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.353045 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.353367 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.353372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.353963 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.354191 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.354193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.355066 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.355067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.355955 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.355957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.356431 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.356432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.356581 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.357427 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.357433 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.359874 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.361956 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.361962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.361965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.362296 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.362509 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.362509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.362510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.362654 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.362983 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.362989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.363182 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.363957 (+ 775us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.364230 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":182,"thread_start_us":55,"threads_started":1}
I20260926 09:22:31.365659 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:35751
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:43935
--webserver_port=39131
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:22:31.365525 32569 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.365767 32568 heartbeater.cc:347] Connected to a master server at 127.24.235.124:33641
I20260926 09:22:31.366283 32568 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.366292 32569 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:31.366364 32568 heartbeater.cc:511] Master 127.24.235.124:33641 requested a full tablet report, sending...
I20260926 09:22:31.366822 31606 ts_manager.cc:194] Re-registered known tserver with Master: 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
I20260926 09:22:31.366914 32389 ts_manager.cc:194] Registered new tserver with Master: 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
I20260926 09:22:31.367271 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:56513
I20260926 09:22:31.367472 32570 heartbeater.cc:347] Connected to a master server at 127.24.235.125:38855
I20260926 09:22:31.367519 32570 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.367592 32570 heartbeater.cc:511] Master 127.24.235.125:38855 requested a full tablet report, sending...
I20260926 09:22:31.367929 32203 ts_manager.cc:194] Registered new tserver with Master: 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
W20260926 09:22:31.450520 32576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:31.450730 32576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:31.450754 32576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:31.451910 32576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:31.451954 32576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:31.451989 32576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:31.453483 32576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:35751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=39131
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:31.453681 32576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:31.453869 32576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:31.458968 32584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.458966 32582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.459172 32576 server_base.cc:1034] running on GCE node
W20260926 09:22:31.459308 32581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.459542 32576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:31.459795 32576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:31.460956 32576 hybrid_clock.cc:648] HybridClock initialized: now 1790414551460942 us; error 28 us; skew 500 ppm
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:31.463508 32576 init.cc:377] Logged in from keytab as oryx/127.24.235.66@KRBTEST.COM (short username oryx)
I20260926 09:22:31.464017 32576 webserver.cc:460] Webserver started at http://127.24.235.66:39131/ using document root <none> and password file <none>
I20260926 09:22:31.464270 32576 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.464330 32576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.465736 32576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:22:31.466357 32591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.466557 32576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:31.466647 32576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "83ca21e2ed2140ca850897b44cb77d85"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "a94586e7d4c54d289a719906c3a8bfcd"
server_key_iv: "d1ad83a604af3ff51bd740bc29e7fc71"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.466996 32576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.489239 32576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:31.491559 32576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:31.491707 32576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:31.491953 32576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:31.492280 32576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:31.492336 32576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.492372 32576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:31.492400 32576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.498481 32576 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:35751
I20260926 09:22:31.498581 32704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:35751 every 8 connection(s)
I20260926 09:22:31.498955 32576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:31.501713 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 32576
I20260926 09:22:31.501869 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31936
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:31.508291 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:43077
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:43935
--webserver_port=35539
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
I20260926 09:22:31.513216 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.500129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:56893 (local address 127.24.235.126:43935)
0926 09:22:31.500252 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.500254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.500264 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:31.501268 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.501269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.501281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.501356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.502483 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.503275 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.504852 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.505197 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.505242 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.508623 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.508625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.508625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.508634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.511675 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.512129 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.512130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.512130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.512192 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.512423 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.512424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.512425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.512579 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.512637 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.512975 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.513097 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 09:22:31.513397 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.500316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:34855 (local address 127.24.235.125:38855)
0926 09:22:31.500399 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.500402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.500413 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:31.501365 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.501367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.501378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.501432 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.503875 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.504688 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.505906 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.506216 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.506289 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.509311 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.509313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.509313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.509322 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.511313 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.511985 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.511987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.511987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.512053 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.512368 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.512369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.512370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.512534 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.512629 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.513191 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.513275 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:22:31.513477 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.500008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:56893)
0926 09:22:31.500174 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.500186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.501144 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.501454 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.501459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.502334 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.502390 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.502391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.503405 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.503406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.504744 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.504746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.505342 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.505343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.505512 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.506227 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.506238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.508500 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.511774 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.511775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.511776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.512040 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.512283 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.512283 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.512284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.512351 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.512665 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.512666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.512797 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.512910 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.513166 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":55,"thread_start_us":58,"threads_started":1}
I20260926 09:22:31.513790 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.500150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.24.235.66:34855)
0926 09:22:31.500320 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.500329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.501154 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.501468 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.501469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.502059 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.502329 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.502335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.504814 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.504815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.505812 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.505815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.506344 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.506344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.506410 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.506794 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.506799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.509203 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.511478 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.511489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.511493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.511896 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.512124 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.512125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.512126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.512299 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.512616 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.512624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.512801 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.513043 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.513667 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":517,"thread_start_us":70,"threads_started":1}
I20260926 09:22:31.514117 32706 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:31.514205 32707 heartbeater.cc:347] Connected to a master server at 127.24.235.125:38855
I20260926 09:22:31.514237 32706 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.514237 32707 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.514470 32707 heartbeater.cc:511] Master 127.24.235.125:38855 requested a full tablet report, sending...
I20260926 09:22:31.514470 32706 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:31.515021 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.499767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:40175 (local address 127.24.235.124:33641)
0926 09:22:31.499812 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.499814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.500041 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:22:31.501489 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.501490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.501507 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.501566 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.502627 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.503338 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.504853 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.505197 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.505242 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.509420 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.509421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.509421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.509429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.511596 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.513744 (+ 2148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.513745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.513745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.513812 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.514227 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.514229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.514229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.514410 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.514508 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.514768 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.514876 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:31.515165 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.499711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.24.235.66:40175)
0926 09:22:31.499875 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.499928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.501143 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.501563 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.501563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.502475 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.502528 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.502530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.503429 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.503430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.504744 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.504746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.505340 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.505342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.505512 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.506914 (+ 1402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.506918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.509316 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.511898 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.511899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.511900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.513647 (+ 1747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.514063 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.514064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.514064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.514132 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.514514 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.514515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.514613 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.514713 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.514833 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":48,"thread_start_us":96,"threads_started":1}
I20260926 09:22:31.515267 32203 ts_manager.cc:194] Registered new tserver with Master: 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751)
I20260926 09:22:31.515543 31606 ts_manager.cc:194] Re-registered known tserver with Master: 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751)
I20260926 09:22:31.516110 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:56893
I20260926 09:22:31.516446 32705 heartbeater.cc:347] Connected to a master server at 127.24.235.124:33641
I20260926 09:22:31.516523 32705 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.516614 32705 heartbeater.cc:511] Master 127.24.235.124:33641 requested a full tablet report, sending...
I20260926 09:22:31.516911 32389 ts_manager.cc:194] Registered new tserver with Master: 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751)
W20260926 09:22:31.597122 32712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:31.597337 32712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:31.597361 32712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:31.598529 32712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:31.598575 32712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:31.598603 32712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:31.600077 32712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:43077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=35539
--tserver_master_addrs=127.24.235.126:43935,127.24.235.125:38855,127.24.235.124:33641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:31.600340 32712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:31.600589 32712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:31.606052 32717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:31.606052 32718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.606326 32712 server_base.cc:1034] running on GCE node
W20260926 09:22:31.606061 32720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:31.606698 32712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:31.606956 32712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:31.608127 32712 hybrid_clock.cc:648] HybridClock initialized: now 1790414551608111 us; error 40 us; skew 500 ppm
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:31.611002 32712 init.cc:377] Logged in from keytab as oryx/127.24.235.67@KRBTEST.COM (short username oryx)
I20260926 09:22:31.611476 32712 webserver.cc:460] Webserver started at http://127.24.235.67:35539/ using document root <none> and password file <none>
I20260926 09:22:31.611779 32712 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:31.611840 32712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:31.613360 32712 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:31.613998 32727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.614199 32712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:31.614276 32712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "08b113f19e954379945e43267c02b20e"
format_stamp: "Formatted at 2026-09-26 09:22:30 on dist-test-slave-7zlw"
server_key: "80cb15a6497ddcaabdba752cb71886de"
server_key_iv: "c0faa3a59261a8487b1a9dc72f086876"
server_key_version: "encryptionkey@0"
I20260926 09:22:31.614619 32712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:31.622639 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.619603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:50820)
0926 09:22:31.619674 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.619689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.619794 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.620031 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.620032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.620034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.620089 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.620091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.620856 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.620857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.621668 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.621670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.622373 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.622374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.622437 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.622441 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.622514 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.622562 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 09:22:31.622639 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.619697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.24.235.124:33641)
0926 09:22:31.619762 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.619764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.619776 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:31.619894 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.619895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.619931 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.620023 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.620175 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.620741 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.621769 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.622267 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.622373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.622456 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.622521 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.622561 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:22:31.623097 32399 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 09:22:31.623274 32428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e68798c8cfff45639b9b0ce71cdff09b. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.623410 32428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: LEARNER
I20260926 09:22:31.628541 32712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:31.630779 32712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:31.630940 32712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:31.631211 32712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:31.631520 32712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:31.631575 32712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.631644 32712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:31.631673 32712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:31.637674 32712 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:43077
I20260926 09:22:31.637735 372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:43077 every 8 connection(s)
I20260926 09:22:31.638087 32712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:31.645527 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 32712
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.125@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.124@KRBTEST.COM
Sep 26 09:22:31 dist-test-slave-7zlw krb5kdc[31561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414551, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:31.652475 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.639199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:35865 (local address 127.24.235.126:43935)
0926 09:22:31.639249 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.639251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.639303 (+ 52us) server_negotiation.cc:407] Connection header received
0926 09:22:31.640378 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.640380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.640396 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.640563 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.643066 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.643743 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.644621 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.644925 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.645067 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.648280 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.648282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.648282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.648291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.650794 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.651315 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.651316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.651316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.651381 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.651675 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.651676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.651676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.651816 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.651882 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.652277 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.652360 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":26}
I20260926 09:22:31.652709 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.639114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:35865)
0926 09:22:31.639240 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.639250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.640262 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.640653 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.640656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.641203 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.641438 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.641443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.643821 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.644546 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.644548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.645048 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.645049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.645119 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.646533 (+ 1414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.646538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.648156 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.650860 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.650867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.650869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.651215 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.651455 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.651456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.651456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.651597 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.651910 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.651916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.652104 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.652241 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.652411 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":6,"thread_start_us":70,"threads_started":1}
I20260926 09:22:31.653255 374 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43935
I20260926 09:22:31.653321 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.638832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:35517 (local address 127.24.235.124:33641)
0926 09:22:31.638891 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.638894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.639098 (+ 204us) server_negotiation.cc:407] Connection header received
0926 09:22:31.640631 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.640633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.640651 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.640716 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.642201 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.642942 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643722 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.644053 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.644131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.647142 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.647144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.647144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.647152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.651539 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.651916 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.651917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.651918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.651974 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.652645 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.652647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.652647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.652793 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.652851 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.653114 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.653203 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:22:31.653363 374 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.653570 374 heartbeater.cc:511] Master 127.24.235.126:43935 requested a full tablet report, sending...
I20260926 09:22:31.654002 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.638824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.24.235.67:35517)
0926 09:22:31.638974 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.639007 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.640254 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.640731 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.640732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.641198 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.641438 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.641443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.643033 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.643634 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.643636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.644141 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.644142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.644203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.644844 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.644849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.647042 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.651616 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.651617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.651618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.651832 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.652052 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.652053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.652053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.652555 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.652899 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.652900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.652946 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.653432 (+ 486us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.653541 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":41,"thread_start_us":85,"threads_started":1}
I20260926 09:22:31.654026 31606 ts_manager.cc:194] Re-registered known tserver with Master: 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:31.654531 31606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:35865
I20260926 09:22:31.654549 373 heartbeater.cc:347] Connected to a master server at 127.24.235.124:33641
I20260926 09:22:31.654599 373 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.654677 373 heartbeater.cc:511] Master 127.24.235.124:33641 requested a full tablet report, sending...
I20260926 09:22:31.655085 32389 ts_manager.cc:194] Registered new tserver with Master: 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:31.655661 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.639355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:54773 (local address 127.24.235.125:38855)
0926 09:22:31.639396 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.639398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.639433 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:22:31.640448 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.640450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.640467 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.640521 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.641563 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.642070 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643507 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.643834 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643903 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.647152 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.647154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.647154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.647161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.649904 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.654806 (+ 4902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.654807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.654807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.654873 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.655085 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.655086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.655087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.655233 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.655288 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.655482 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.655559 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:31.656049 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.639283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.24.235.67:54773)
0926 09:22:31.639387 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.639396 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.640275 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.640655 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.640658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.641198 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:31.641438 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.641444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.642184 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.642185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.643410 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.643412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.643926 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.643927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.644067 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.644675 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:31.644683 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:31.647028 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:31.653602 (+ 6574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.653603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.653603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.653827 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.654957 (+ 1130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:31.654958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:31.654958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:31.655014 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:31.655324 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:31.655324 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:31.655367 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.655779 (+ 412us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.655908 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":23,"thread_start_us":74,"threads_started":1}
I20260926 09:22:31.656371 375 heartbeater.cc:347] Connected to a master server at 127.24.235.125:38855
I20260926 09:22:31.656421 375 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:31.656481 375 heartbeater.cc:511] Master 127.24.235.125:38855 requested a full tablet report, sending...
I20260926 09:22:31.656746 32203 ts_manager.cc:194] Registered new tserver with Master: 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:31.656760 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.647317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.24.235.126:43935)
0926 09:22:31.648440 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.648442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.648575 (+ 133us) server_negotiation.cc:407] Connection header received
0926 09:22:31.648642 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.648643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.648659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.648742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.649542 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.650353 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.651455 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.651799 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.651863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.653594 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.653596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.653596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.653617 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.655382 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.655767 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.655769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.655769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.655823 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.656033 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.656033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.656034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.656158 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.656251 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.656588 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.656678 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1092,"thread_start_us":1067,"threads_started":1}
I20260926 09:22:31.659663 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.647894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.24.235.124:33641)
0926 09:22:31.648690 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.648692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.649045 (+ 353us) server_negotiation.cc:407] Connection header received
0926 09:22:31.649069 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.649071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.649084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.649148 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.649866 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.650589 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.651636 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.651926 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.651990 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.654757 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.654758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.654759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.654766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.658346 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.658802 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.658803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.658803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.658864 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.659100 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.659101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.659101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.659222 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.659343 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.659453 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.659540 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":67,"threads_started":1}
I20260926 09:22:31.658753 31606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:31.661185 31606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:31.661715 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.648185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.24.235.125:38855)
0926 09:22:31.648354 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.648361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.648796 (+ 435us) server_negotiation.cc:407] Connection header received
0926 09:22:31.648872 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.648873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.648888 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.648966 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.649751 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.650521 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.657042 (+ 6521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.657334 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.657420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.658667 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.658669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.658669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.658677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.660348 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.660769 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.660770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.660770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.660833 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.661072 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.661073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.661073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.661211 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.661309 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.661493 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.661577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260926 09:22:31.662050 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: NON_VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: true } } }
I20260926 09:22:31.662159 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.663514 32164 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: attempting to promote NON_VOTER 99213f8260384145a89fd03e3266fe94 to VOTER
I20260926 09:22:31.663746 32164 consensus_queue.cc:237] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } }
I20260926 09:22:31.664176 32213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Refusing update from remote peer e68798c8cfff45639b9b0ce71cdff09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 09:22:31.664256 32399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 LEARNER]: Refusing update from remote peer e68798c8cfff45639b9b0ce71cdff09b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 09:22:31.664390 32164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 09:22:31.664515 32353 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 09:22:31.665323 32164 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99213f8260384145a89fd03e3266fe94 (127.24.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.665563 32399 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99213f8260384145a89fd03e3266fe94 (127.24.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.666409 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.666538 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.666599 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.666587 32213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99213f8260384145a89fd03e3266fe94 (127.24.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.666692 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.666689 32428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.666802 32428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:22:31.667666 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.667750 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:22:31.666429 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e68798c8cfff45639b9b0ce71cdff09b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:31.667985 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:31.673224 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.668720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40347 (local address 127.0.0.1:39286)
0926 09:22:31.669002 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.669037 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.669082 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.669640 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.669641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.669643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.669693 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.669695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.671144 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.671145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.672192 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.672194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.672909 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.672911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.672967 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.672971 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.673089 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.673146 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":29}
I20260926 09:22:31.673235 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.668838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.24.235.65:40347)
0926 09:22:31.669394 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.669399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.669411 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:31.669473 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.669475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.669495 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.669817 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.669882 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.671060 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.672278 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.672812 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.672912 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.673000 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.673094 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.673136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":101,"threads_started":1}
I20260926 09:22:31.673416 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.668815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43077 (local address 127.0.0.1:56936)
0926 09:22:31.669209 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.669219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.669274 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.669606 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.669607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.669609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.669678 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.669680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.670726 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.670728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.671975 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.671978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.673152 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.673154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.673219 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.673223 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.673271 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.673315 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260926 09:22:31.673593 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.668822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35751 (local address 127.0.0.1:48618)
0926 09:22:31.669431 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.669440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.669499 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.669786 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.669787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.669789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.669831 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.669833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.671110 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.671112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.672290 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.672294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.673377 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.673378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.673433 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.673437 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.673487 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.673532 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":38,"thread_start_us":106,"threads_started":1}
I20260926 09:22:31.673699 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.669257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.24.235.67:43077)
0926 09:22:31.669418 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.669421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.669430 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:31.669487 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.669489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.669506 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.669609 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.669736 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.670628 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.672089 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.672929 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.673005 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.673114 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.673554 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.673601 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260926 09:22:31.674105 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.668942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.24.235.66:35751)
0926 09:22:31.669188 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.669194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.669559 (+ 365us) server_negotiation.cc:407] Connection header received
0926 09:22:31.669632 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.669635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.669650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.669720 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.669908 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.671009 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.672404 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.673212 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.673280 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.673344 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.673952 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.673995 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":163,"threads_started":1}
I20260926 09:22:31.674479 307 tablet_service.cc:1467] Processing CreateTablet for tablet d511f450aa6146d19e67e97efa21d054 (DEFAULT_TABLE table=test-table [id=9b9dc3f37cfa45f9866a12a1670e6148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:31.674528 32502 tablet_service.cc:1467] Processing CreateTablet for tablet d511f450aa6146d19e67e97efa21d054 (DEFAULT_TABLE table=test-table [id=9b9dc3f37cfa45f9866a12a1670e6148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:31.674863 307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d511f450aa6146d19e67e97efa21d054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:31.674873 32502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d511f450aa6146d19e67e97efa21d054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:31.677533 32639 tablet_service.cc:1467] Processing CreateTablet for tablet d511f450aa6146d19e67e97efa21d054 (DEFAULT_TABLE table=test-table [id=9b9dc3f37cfa45f9866a12a1670e6148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:31.677922 32639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d511f450aa6146d19e67e97efa21d054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:31.678062 402 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Bootstrap starting.
I20260926 09:22:31.678164 403 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Bootstrap starting.
I20260926 09:22:31.678857 403 tablet_bootstrap.cc:654] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:31.679114 402 tablet_bootstrap.cc:654] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:31.679195 403 log.cc:824] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:31.679545 402 log.cc:824] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:31.680019 403 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: No bootstrap required, opened a new log
I20260926 09:22:31.680099 403 ts_tablet_manager.cc:1397] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:31.680317 402 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: No bootstrap required, opened a new log
I20260926 09:22:31.680385 402 ts_tablet_manager.cc:1397] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:31.682823 403 raft_consensus.cc:348] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.683015 403 raft_consensus.cc:374] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:31.683054 403 raft_consensus.cc:729] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b026bd0a746f09ab0f03faf2f7b0d, State: Initialized, Role: FOLLOWER
I20260926 09:22:31.683104 402 raft_consensus.cc:348] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.683184 403 consensus_queue.cc:260] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.683267 402 raft_consensus.cc:374] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:31.683305 402 raft_consensus.cc:729] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b113f19e954379945e43267c02b20e, State: Initialized, Role: FOLLOWER
I20260926 09:22:31.683419 402 consensus_queue.cc:260] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.683457 403 ts_tablet_manager.cc:1428] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:22:31.683521 32569 heartbeater.cc:503] Master 127.24.235.126:43935 was elected leader, sending a full tablet report...
I20260926 09:22:31.683845 374 heartbeater.cc:503] Master 127.24.235.126:43935 was elected leader, sending a full tablet report...
I20260926 09:22:31.684312 402 ts_tablet_manager.cc:1428] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:22:31.684562 406 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Bootstrap starting.
I20260926 09:22:31.685288 406 tablet_bootstrap.cc:654] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:31.685628 406 log.cc:824] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:31.686399 406 tablet_bootstrap.cc:492] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: No bootstrap required, opened a new log
I20260926 09:22:31.686456 406 ts_tablet_manager.cc:1397] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:31.688158 406 raft_consensus.cc:348] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.688297 406 raft_consensus.cc:374] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:31.688318 406 raft_consensus.cc:729] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ca21e2ed2140ca850897b44cb77d85, State: Initialized, Role: FOLLOWER
I20260926 09:22:31.688464 406 consensus_queue.cc:260] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.688756 406 ts_tablet_manager.cc:1428] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:22:31.688787 32706 heartbeater.cc:503] Master 127.24.235.126:43935 was elected leader, sending a full tablet report...
I20260926 09:22:31.705868 410 raft_consensus.cc:482] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:31.706014 410 raft_consensus.cc:504] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.706322 410 leader_election.cc:288] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347), 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:31.710234 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.706660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:38353 (local address 127.24.235.65:40347)
0926 09:22:31.706719 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.706721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.706737 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:22:31.706770 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.706771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.706782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.706866 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.707076 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.707865 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.709458 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.709834 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.709893 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.709899 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.710085 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.710131 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:22:31.710233 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.706542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40347 (local address 127.24.235.66:38353)
0926 09:22:31.706648 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.706661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.706709 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.706879 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.706879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.706881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.706993 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.706994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.707939 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.707940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.709348 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.709351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.709912 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.709913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.710005 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.710009 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.710065 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.710114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":67}
I20260926 09:22:31.710428 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.706767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:57509 (local address 127.24.235.67:43077)
0926 09:22:31.706811 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.706813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.706820 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:22:31.706844 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.706845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.706853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.706909 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:31.707075 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.707820 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.709452 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.709932 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.710006 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.710014 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.710291 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.710342 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:31.710584 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.706685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43077 (local address 127.24.235.66:57509)
0926 09:22:31.706729 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:31.706738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:31.706767 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:31.706926 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:31.706927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:31.706930 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:31.706992 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.706994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.707939 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.707940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.709348 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:31.709351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.710027 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.710028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:31.710141 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.710145 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:31.710202 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 09:22:31.710238 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":117}
I20260926 09:22:31.710685 32521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d511f450aa6146d19e67e97efa21d054" candidate_uuid: "83ca21e2ed2140ca850897b44cb77d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" is_pre_election: true
I20260926 09:22:31.710903 32521 raft_consensus.cc:2454] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ca21e2ed2140ca850897b44cb77d85 in term 0.
I20260926 09:22:31.710939 327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d511f450aa6146d19e67e97efa21d054" candidate_uuid: "83ca21e2ed2140ca850897b44cb77d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b113f19e954379945e43267c02b20e" is_pre_election: true
I20260926 09:22:31.711068 327 raft_consensus.cc:2454] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ca21e2ed2140ca850897b44cb77d85 in term 0.
I20260926 09:22:31.711193 32592 leader_election.cc:302] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8b026bd0a746f09ab0f03faf2f7b0d, 83ca21e2ed2140ca850897b44cb77d85; no voters:
I20260926 09:22:31.711366 410 raft_consensus.cc:2781] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:31.711450 410 raft_consensus.cc:482] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:31.711479 410 raft_consensus.cc:3037] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:31.712345 410 raft_consensus.cc:504] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.712489 410 leader_election.cc:288] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [CANDIDATE]: Term 1 election: Requested vote from peers 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347), 08b113f19e954379945e43267c02b20e (127.24.235.67:43077)
I20260926 09:22:31.712719 32521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d511f450aa6146d19e67e97efa21d054" candidate_uuid: "83ca21e2ed2140ca850897b44cb77d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
I20260926 09:22:31.712733 327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d511f450aa6146d19e67e97efa21d054" candidate_uuid: "83ca21e2ed2140ca850897b44cb77d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b113f19e954379945e43267c02b20e"
I20260926 09:22:31.712806 32521 raft_consensus.cc:3037] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:31.712822 327 raft_consensus.cc:3037] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:31.713742 32521 raft_consensus.cc:2454] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ca21e2ed2140ca850897b44cb77d85 in term 1.
I20260926 09:22:31.713742 327 raft_consensus.cc:2454] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ca21e2ed2140ca850897b44cb77d85 in term 1.
I20260926 09:22:31.713894 32592 leader_election.cc:302] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b113f19e954379945e43267c02b20e, 83ca21e2ed2140ca850897b44cb77d85; no voters:
I20260926 09:22:31.713986 410 raft_consensus.cc:2781] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:31.714166 410 raft_consensus.cc:686] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 LEADER]: Becoming Leader. State: Replica: 83ca21e2ed2140ca850897b44cb77d85, State: Running, Role: LEADER
I20260926 09:22:31.714282 410 consensus_queue.cc:237] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } }
I20260926 09:22:31.715271 31606 catalog_manager.cc:5495] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "83ca21e2ed2140ca850897b44cb77d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } health_report { overall_health: HEALTHY } } }
I20260926 09:22:31.723448 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.720965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.24.235.66:35751)
0926 09:22:31.721033 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.721035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.721044 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:31.721100 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.721101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.721123 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.721195 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:31.721330 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.721805 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.722626 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.722847 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.722907 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.723072 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:31.723208 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:31.723313 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.723348 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.723372 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:22:31.725340 327 raft_consensus.cc:1261] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Refusing update from remote peer 83ca21e2ed2140ca850897b44cb77d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:31.725340 32521 raft_consensus.cc:1261] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Refusing update from remote peer 83ca21e2ed2140ca850897b44cb77d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:31.725745 411 consensus_queue.cc:1035] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:31.725912 410 consensus_queue.cc:1035] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:31.729666 416 mvcc.cc:205] Tried to move back new op lower bound from 7333538003864002560 to 7333538003822534656. Current Snapshot: MvccSnapshot[applied={T|T < 7333538003864002560}]
I20260926 09:22:31.729977 415 mvcc.cc:205] Tried to move back new op lower bound from 7333538003864002560 to 7333538003822534656. Current Snapshot: MvccSnapshot[applied={T|T < 7333538003864002560}]
I20260926 09:22:31.731163 31606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:31.731283 31606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:31.733028 31606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e68798c8cfff45639b9b0ce71cdff09b: Sending DeleteTablet for 3 replicas of tablet d511f450aa6146d19e67e97efa21d054
I20260926 09:22:31.733312 32502 tablet_service.cc:1514] Processing DeleteTablet for tablet d511f450aa6146d19e67e97efa21d054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:31 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:39286
I20260926 09:22:31.733371 307 tablet_service.cc:1514] Processing DeleteTablet for tablet d511f450aa6146d19e67e97efa21d054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:31 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:56936
I20260926 09:22:31.733606 424 tablet_replica.cc:330] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: stopping tablet replica
I20260926 09:22:31.733721 424 raft_consensus.cc:2229] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:31.733821 424 raft_consensus.cc:2258] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:31.733955 32639 tablet_service.cc:1514] Processing DeleteTablet for tablet d511f450aa6146d19e67e97efa21d054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:31 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:48618
I20260926 09:22:31.734357 424 ts_tablet_manager.cc:1905] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:31.734573 425 tablet_replica.cc:330] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: stopping tablet replica
I20260926 09:22:31.734726 425 raft_consensus.cc:2229] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:31.734898 425 raft_consensus.cc:2258] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:31.735175 423 tablet_replica.cc:330] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: stopping tablet replica
I20260926 09:22:31.735313 423 raft_consensus.cc:2229] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:31.735453 423 raft_consensus.cc:2258] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:31.735678 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 31574
I20260926 09:22:31.736040 423 ts_tablet_manager.cc:1905] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:31.736256 424 ts_tablet_manager.cc:1918] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:31.736385 424 log.cc:1196] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/d511f450aa6146d19e67e97efa21d054
I20260926 09:22:31.736722 424 ts_tablet_manager.cc:1939] T d511f450aa6146d19e67e97efa21d054 P 08b113f19e954379945e43267c02b20e: Deleting consensus metadata
I20260926 09:22:31.737975 423 ts_tablet_manager.cc:1918] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:31.738066 423 log.cc:1196] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/d511f450aa6146d19e67e97efa21d054
I20260926 09:22:31.738320 423 ts_tablet_manager.cc:1939] T d511f450aa6146d19e67e97efa21d054 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Deleting consensus metadata
I20260926 09:22:31.741518 425 ts_tablet_manager.cc:1905] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:22:31.742906 32729 connection.cc:535] server connection from 127.0.0.1:56936 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:22:31.743091 32457 connection.cc:535] server connection from 127.0.0.1:39286 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:22:31.743335 425 ts_tablet_manager.cc:1918] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:31.743433 425 log.cc:1196] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/d511f450aa6146d19e67e97efa21d054
I20260926 09:22:31.743707 425 ts_tablet_manager.cc:1939] T d511f450aa6146d19e67e97efa21d054 P 83ca21e2ed2140ca850897b44cb77d85: Deleting consensus metadata
W20260926 09:22:31.744040 32592 connection.cc:441] server connection from 127.0.0.1:48618 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48618 (request call id 1) could send its response
I20260926 09:22:31.753352 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.734926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.24.235.124:33641)
0926 09:22:31.734967 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.734969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.744170 (+ 9201us) server_negotiation.cc:407] Connection header received
0926 09:22:31.744295 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.744296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.744311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.744376 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.745131 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.745961 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.747071 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.747398 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.747439 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.749590 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.749591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.749592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.749599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.752005 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752405 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.752460 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752690 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.752834 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.752914 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.753145 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.753239 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:31.753593 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.745478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.24.235.125:38855)
0926 09:22:31.745571 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.745573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.745581 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:31.745603 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.745603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.745616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.745665 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.746899 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.747627 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.748598 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.748907 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.748949 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.750359 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.750360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.750360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.750368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.752291 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752725 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.752783 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752974 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.753125 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.753205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.753408 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.753496 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 09:22:31.753862 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:31.745524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.24.235.124:33641)
0926 09:22:31.745565 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:31.745567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:31.745575 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:31.745597 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:31.745598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:31.745615 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:31.745665 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:31.746637 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.747379 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.748427 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:31.748724 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:31.748761 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:31.750358 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:31.750359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:31.750360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:31.750368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:31.752198 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752598 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.752645 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:31.752839 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:31.752840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:31.752841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:31.752939 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:31.752999 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:31.753229 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 09:22:31.753289 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:32.731964 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:32.731661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:33075)
0926 09:22:32.731833 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:32.731919 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
W20260926 09:22:32.732257 32706 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:43935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:32.733899 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:32.733648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:36231)
0926 09:22:32.733782 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:32.733861 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260926 09:22:32.734107 32569 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:43935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:32.734979 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:32.734751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:53101)
0926 09:22:32.734870 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:32.734937 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
W20260926 09:22:32.735154 374 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:43935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:33.233752 436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e68798c8cfff45639b9b0ce71cdff09b)
I20260926 09:22:33.233898 437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e68798c8cfff45639b9b0ce71cdff09b)
I20260926 09:22:33.233946 436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } }
I20260926 09:22:33.234056 437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } }
I20260926 09:22:33.234301 436 leader_election.cc:288] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935), 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125:38855)
I20260926 09:22:33.234352 437 leader_election.cc:288] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935), 99213f8260384145a89fd03e3266fe94 (127.24.235.124:33641)
I20260926 09:22:33.234867 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47148)
0926 09:22:33.234768 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.234822 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":53,"threads_started":1}
I20260926 09:22:33.234901 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47158)
0926 09:22:33.234816 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.234864 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":92,"threads_started":1}
W20260926 09:22:33.234982 32188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
W20260926 09:22:33.234972 32374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:33.235410 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.235329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47166)
0926 09:22:33.235363 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.235387 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 09:22:33.235502 32188 leader_election.cc:334] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935): Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:33.235591 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.235361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47172)
0926 09:22:33.235398 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.235425 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 09:22:33.235674 32374 leader_election.cc:334] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935): Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:33.238067 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:38855 (local address 127.0.0.1:36612)
0926 09:22:33.234993 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.235004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.235057 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.235310 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.235311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.235313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.235370 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.235372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.236155 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.236156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.236990 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.236993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.237768 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.237769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.237866 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.237871 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.237942 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.237992 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":66,"threads_started":1}
I20260926 09:22:33.238073 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.24.235.125:38855)
0926 09:22:33.235145 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.235147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.235154 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:22:33.235179 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.235180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.235191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.235256 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.235447 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.236072 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.237104 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.237698 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.237770 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.237779 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.237976 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.238017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":53,"threads_started":1}
I20260926 09:22:33.238268 439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:33641 (local address 127.0.0.1:53114)
0926 09:22:33.234830 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.234841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.234898 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.235371 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.235372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.235375 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.235444 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.235447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.236263 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.236264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.237190 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.237192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.238018 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.238019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.238078 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.238083 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.238125 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.238177 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":40,"threads_started":1}
I20260926 09:22:33.238338 32213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99213f8260384145a89fd03e3266fe94" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "10743bbd99de4fb7bcbf99d6336c5024" is_pre_election: true
I20260926 09:22:33.238464 32213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99213f8260384145a89fd03e3266fe94 in term 1.
I20260926 09:22:33.238584 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.234885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.24.235.124:33641)
0926 09:22:33.235202 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.235204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.235214 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:33.235242 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.235243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.235254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.235519 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.235555 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.236171 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.237279 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.237907 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.238436 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.238446 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.238464 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.238507 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":51,"threads_started":1}
I20260926 09:22:33.238615 32374 leader_election.cc:302] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10743bbd99de4fb7bcbf99d6336c5024, 99213f8260384145a89fd03e3266fe94; no voters: e68798c8cfff45639b9b0ce71cdff09b
I20260926 09:22:33.238765 436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 09:22:33.238826 436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e68798c8cfff45639b9b0ce71cdff09b)
I20260926 09:22:33.238811 32399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10743bbd99de4fb7bcbf99d6336c5024" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "99213f8260384145a89fd03e3266fe94" is_pre_election: true
I20260926 09:22:33.238899 436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:22:33.239639 436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } }
I20260926 09:22:33.239811 436 leader_election.cc:288] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 election: Requested vote from peers e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935), 10743bbd99de4fb7bcbf99d6336c5024 (127.24.235.125:38855)
I20260926 09:22:33.239908 32399 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10743bbd99de4fb7bcbf99d6336c5024 in current term 2: Already voted for candidate 99213f8260384145a89fd03e3266fe94 in this term.
I20260926 09:22:33.240054 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.239942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47178)
0926 09:22:33.239994 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.240027 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 09:22:33.240147 32213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99213f8260384145a89fd03e3266fe94" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "10743bbd99de4fb7bcbf99d6336c5024"
I20260926 09:22:33.240267 32213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:22:33.240360 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.240280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47186)
0926 09:22:33.240314 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.240336 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 09:22:33.240413 32374 leader_election.cc:334] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935): Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
I20260926 09:22:33.241122 32213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99213f8260384145a89fd03e3266fe94 in term 2.
I20260926 09:22:33.241295 32374 leader_election.cc:302] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10743bbd99de4fb7bcbf99d6336c5024, 99213f8260384145a89fd03e3266fe94; no voters: e68798c8cfff45639b9b0ce71cdff09b
I20260926 09:22:33.241365 32188 leader_election.cc:302] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10743bbd99de4fb7bcbf99d6336c5024; no voters: 99213f8260384145a89fd03e3266fe94, e68798c8cfff45639b9b0ce71cdff09b
I20260926 09:22:33.241406 436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:22:33.241536 436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [term 2 LEADER]: Becoming Leader. State: Replica: 99213f8260384145a89fd03e3266fe94, State: Running, Role: LEADER
I20260926 09:22:33.241525 437 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:22:33.241612 436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } }
I20260926 09:22:33.241955 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99213f8260384145a89fd03e3266fe94. Latest consensus state: current_term: 2 leader_uuid: "99213f8260384145a89fd03e3266fe94" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:33.242074 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:33.242233 448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:33.242645 448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:33.242710 448 catalog_manager.cc:1259] Loaded cluster ID: 6bfa86a02d4941d9a2e7a94a2e69b98a
I20260926 09:22:33.242738 448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:33.242883 448 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:33.242947 448 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Loaded TSK: 0
I20260926 09:22:33.243054 448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:33.246395 32389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:33.248113 32389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:33.249240 32213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [term 2 FOLLOWER]: Refusing update from remote peer 99213f8260384145a89fd03e3266fe94: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 09:22:33.249290 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.249194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47204)
0926 09:22:33.249238 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.249268 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 09:22:33.249518 436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 09:22:33.249755 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.249620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.0.0.1:47206)
0926 09:22:33.249694 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.249729 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 09:22:33.249853 32374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 -> Peer e68798c8cfff45639b9b0ce71cdff09b (127.24.235.126:43935): Couldn't send request to peer e68798c8cfff45639b9b0ce71cdff09b. Status: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 09:22:33.250275 437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99213f8260384145a89fd03e3266fe94. Latest consensus state: current_term: 2 leader_uuid: "99213f8260384145a89fd03e3266fe94" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:33.250358 437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:22:33.250924 436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "99213f8260384145a89fd03e3266fe94" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:33.251041 436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:33.251008 437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "99213f8260384145a89fd03e3266fe94" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e68798c8cfff45639b9b0ce71cdff09b" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43935 } } peers { permanent_uuid: "10743bbd99de4fb7bcbf99d6336c5024" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 38855 } attrs { promote: false } } peers { permanent_uuid: "99213f8260384145a89fd03e3266fe94" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 33641 } attrs { promote: false } } }
I20260926 09:22:33.251120 437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10743bbd99de4fb7bcbf99d6336c5024 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:22:33.256418 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.252996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.24.235.65:40347)
0926 09:22:33.253234 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.253236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.253495 (+ 259us) server_negotiation.cc:407] Connection header received
0926 09:22:33.253523 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.253524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.253534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.253590 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.253852 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.255631 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256051 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.256152 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.256292 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.256335 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260926 09:22:33.256425 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.252929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40347 (local address 127.0.0.1:35928)
0926 09:22:33.253396 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.253405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.253444 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.253748 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.253748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.253749 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.253778 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.253779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254655 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.254656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.255555 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.255557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256135 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.256197 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.256202 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.256283 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.256327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":75,"threads_started":1}
I20260926 09:22:33.256431 454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.252867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35751 (local address 127.0.0.1:56316)
0926 09:22:33.253071 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.253082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.253133 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.253457 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.253457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.253459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.253534 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.253536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254491 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.254491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.255324 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.255327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256127 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.256195 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.256200 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.256231 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.256286 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":5}
I20260926 09:22:33.256667 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.253127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.24.235.67:43077)
0926 09:22:33.253266 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.253269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.253277 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:33.253307 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.253308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.253318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.253524 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.253598 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254086 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.255236 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256093 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256400 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.256416 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.256435 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.256479 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260926 09:22:33.256683 32502 tablet_service.cc:1467] Processing CreateTablet for tablet cef85debdd9b4023aaeb9f369674eac1 (DEFAULT_TABLE table=test-table [id=82de44359d8f4195bea574acdb5cd4f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:33.256810 32502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef85debdd9b4023aaeb9f369674eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:33.256910 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.252827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43077 (local address 127.0.0.1:39686)
0926 09:22:33.252955 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.252967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.253018 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.253431 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.253432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.253436 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.253536 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.253538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254166 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.254167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.255138 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.255140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256176 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.256232 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.256235 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.256262 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.256297 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":38}
I20260926 09:22:33.257205 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.253178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.24.235.66:35751)
0926 09:22:33.253313 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.253315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.253325 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:33.253351 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.253352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.253364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.253615 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.253655 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.254401 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.255444 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.256032 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.256449 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.257041 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.257066 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.257120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260926 09:22:33.257323 307 tablet_service.cc:1467] Processing CreateTablet for tablet cef85debdd9b4023aaeb9f369674eac1 (DEFAULT_TABLE table=test-table [id=82de44359d8f4195bea574acdb5cd4f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:33.257376 32639 tablet_service.cc:1467] Processing CreateTablet for tablet cef85debdd9b4023aaeb9f369674eac1 (DEFAULT_TABLE table=test-table [id=82de44359d8f4195bea574acdb5cd4f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:33.257441 307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef85debdd9b4023aaeb9f369674eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:33.257468 32639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef85debdd9b4023aaeb9f369674eac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:33.258285 458 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Bootstrap starting.
I20260926 09:22:33.258720 458 tablet_bootstrap.cc:654] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:33.259068 459 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Bootstrap starting.
I20260926 09:22:33.259475 459 tablet_bootstrap.cc:654] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:33.259768 460 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Bootstrap starting.
I20260926 09:22:33.260128 460 tablet_bootstrap.cc:654] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:33.260156 459 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: No bootstrap required, opened a new log
I20260926 09:22:33.260193 459 ts_tablet_manager.cc:1397] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:33.260342 459 raft_consensus.cc:348] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.260433 459 raft_consensus.cc:374] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:33.260466 459 raft_consensus.cc:729] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b113f19e954379945e43267c02b20e, State: Initialized, Role: FOLLOWER
I20260926 09:22:33.260511 459 consensus_queue.cc:260] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.260727 459 ts_tablet_manager.cc:1428] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:33.260740 460 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: No bootstrap required, opened a new log
I20260926 09:22:33.260789 460 ts_tablet_manager.cc:1397] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:33.260888 460 raft_consensus.cc:348] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.260957 460 raft_consensus.cc:374] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:33.260972 460 raft_consensus.cc:729] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ca21e2ed2140ca850897b44cb77d85, State: Initialized, Role: FOLLOWER
I20260926 09:22:33.261047 460 consensus_queue.cc:260] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.261215 460 ts_tablet_manager.cc:1428] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:33.261641 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.261395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:37401)
0926 09:22:33.261548 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.261602 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 09:22:33.261860 458 tablet_bootstrap.cc:492] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d: No bootstrap required, opened a new log
I20260926 09:22:33.261910 458 ts_tablet_manager.cc:1397] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 09:22:33.262048 458 raft_consensus.cc:348] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.262140 458 raft_consensus.cc:374] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:33.262166 458 raft_consensus.cc:729] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b026bd0a746f09ab0f03faf2f7b0d, State: Initialized, Role: FOLLOWER
I20260926 09:22:33.262223 458 consensus_queue.cc:260] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.262351 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.261395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:56323)
0926 09:22:33.262087 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.262310 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":87,"threads_started":1}
I20260926 09:22:33.262408 458 ts_tablet_manager.cc:1428] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:33.262159 373 heartbeater.cc:503] Master 127.24.235.124:33641 was elected leader, sending a full tablet report...
I20260926 09:22:33.262354 32705 heartbeater.cc:503] Master 127.24.235.124:33641 was elected leader, sending a full tablet report...
I20260926 09:22:33.262974 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.262849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:40575)
0926 09:22:33.262911 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.262942 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:22:33.263062 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.262809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:57757)
0926 09:22:33.262991 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.263029 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260926 09:22:33.263094 32568 heartbeater.cc:503] Master 127.24.235.124:33641 was elected leader, sending a full tablet report...
I20260926 09:22:33.263715 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.263625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:55899)
0926 09:22:33.263657 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.263684 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 09:22:33.263720 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.263528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:37979)
0926 09:22:33.263676 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.263700 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 09:22:33.270826 464 raft_consensus.cc:482] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:33.270900 464 raft_consensus.cc:504] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.271214 464 leader_election.cc:288] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751), 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
I20260926 09:22:33.274978 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.271336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35751 (local address 127.24.235.67:52177)
0926 09:22:33.271385 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.271397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.271444 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.271577 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.271578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.271580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.271704 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.271707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.272659 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.272660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.273933 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.273936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.274687 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.274783 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.274787 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.274846 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.274888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260926 09:22:33.275012 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.271376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:52177 (local address 127.24.235.66:35751)
0926 09:22:33.271447 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.271449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.271464 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:22:33.271493 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.271494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.271503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.271569 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.271784 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.272578 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274036 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.274606 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274692 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.274705 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.274885 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.274933 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:22:33.275079 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.271678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:46295 (local address 127.24.235.65:40347)
0926 09:22:33.271742 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.271748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.271758 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:33.271817 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.271818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.271828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.271883 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:33.272006 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.272676 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274036 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.274660 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274724 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.274737 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.274959 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.275007 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:22:33.275403 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.271438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40347 (local address 127.24.235.67:46295)
0926 09:22:33.271693 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.271704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.271748 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.271893 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:33.271893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:33.271894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:33.271941 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.271942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.272746 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.272746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.273933 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:33.273936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.274752 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.274754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:33.274810 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.274813 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:33.275270 (+ 457us) client_negotiation.cc:240] Negotiation successful
0926 09:22:33.275318 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":2,"thread_start_us":192,"threads_started":1}
I20260926 09:22:33.275334 32659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef85debdd9b4023aaeb9f369674eac1" candidate_uuid: "08b113f19e954379945e43267c02b20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ca21e2ed2140ca850897b44cb77d85" is_pre_election: true
I20260926 09:22:33.275506 32659 raft_consensus.cc:2454] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b113f19e954379945e43267c02b20e in term 0.
I20260926 09:22:33.275691 32729 leader_election.cc:302] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b113f19e954379945e43267c02b20e, 83ca21e2ed2140ca850897b44cb77d85; no voters:
I20260926 09:22:33.275739 32521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef85debdd9b4023aaeb9f369674eac1" candidate_uuid: "08b113f19e954379945e43267c02b20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" is_pre_election: true
I20260926 09:22:33.275844 464 raft_consensus.cc:2781] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:33.275851 32521 raft_consensus.cc:2454] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b113f19e954379945e43267c02b20e in term 0.
I20260926 09:22:33.275883 464 raft_consensus.cc:482] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:33.275915 464 raft_consensus.cc:3037] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:33.276656 464 raft_consensus.cc:504] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.276798 464 leader_election.cc:288] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [CANDIDATE]: Term 1 election: Requested vote from peers 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751), 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347)
I20260926 09:22:33.276937 32521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef85debdd9b4023aaeb9f369674eac1" candidate_uuid: "08b113f19e954379945e43267c02b20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d"
I20260926 09:22:33.277009 32521 raft_consensus.cc:3037] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:33.276968 32659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef85debdd9b4023aaeb9f369674eac1" candidate_uuid: "08b113f19e954379945e43267c02b20e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ca21e2ed2140ca850897b44cb77d85"
I20260926 09:22:33.277099 32659 raft_consensus.cc:3037] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:33.277626 32521 raft_consensus.cc:2454] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b113f19e954379945e43267c02b20e in term 1.
I20260926 09:22:33.277827 32728 leader_election.cc:302] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b113f19e954379945e43267c02b20e, 0a8b026bd0a746f09ab0f03faf2f7b0d; no voters:
I20260926 09:22:33.277921 464 raft_consensus.cc:2781] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:33.277922 32659 raft_consensus.cc:2454] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b113f19e954379945e43267c02b20e in term 1.
I20260926 09:22:33.278082 464 raft_consensus.cc:686] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 1 LEADER]: Becoming Leader. State: Replica: 08b113f19e954379945e43267c02b20e, State: Running, Role: LEADER
I20260926 09:22:33.278156 464 consensus_queue.cc:237] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } }
I20260926 09:22:33.278358 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.278259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:45591)
0926 09:22:33.278293 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.278327 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 09:22:33.278895 32389 catalog_manager.cc:5495] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e reported cstate change: term changed from 0 to 1, leader changed from <none> to 08b113f19e954379945e43267c02b20e (127.24.235.67). New cstate: current_term: 1 leader_uuid: "08b113f19e954379945e43267c02b20e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b113f19e954379945e43267c02b20e" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:33.285926 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.283426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.24.235.67:43077)
0926 09:22:33.283472 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:33.283475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:33.283485 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:33.283603 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:33.283603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:33.283620 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:33.283696 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:33.283843 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.284328 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.285127 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:33.285318 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:33.285401 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:33.285580 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:33.285693 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:33.285780 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:33.285820 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:22:33.285840 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:22:33.287109 32521 raft_consensus.cc:1261] T cef85debdd9b4023aaeb9f369674eac1 P 0a8b026bd0a746f09ab0f03faf2f7b0d [term 1 FOLLOWER]: Refusing update from remote peer 08b113f19e954379945e43267c02b20e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:33.287184 32659 raft_consensus.cc:1261] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Refusing update from remote peer 08b113f19e954379945e43267c02b20e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:33.287405 464 consensus_queue.cc:1035] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8b026bd0a746f09ab0f03faf2f7b0d" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:33.287601 464 consensus_queue.cc:1035] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ca21e2ed2140ca850897b44cb77d85" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:33.288439 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.288320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:50277)
0926 09:22:33.288363 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.288405 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:22:33.289191 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.289036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:42533)
0926 09:22:33.289126 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.289157 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 09:22:33.289608 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.289501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:32979)
0926 09:22:33.289542 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.289576 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:22:33.290161 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.290071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.65:56617)
0926 09:22:33.290108 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.290132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 09:22:33.291034 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.290944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.67:40409)
0926 09:22:33.290977 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.291004 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:22:33.291786 32389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:33.291838 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:33.291636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43935 (local address 127.24.235.66:38885)
0926 09:22:33.291783 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.291812 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 09:22:33.291882 32389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:33.293282 32389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99213f8260384145a89fd03e3266fe94: Sending DeleteTablet for 3 replicas of tablet cef85debdd9b4023aaeb9f369674eac1
I20260926 09:22:33.293527 32502 tablet_service.cc:1514] Processing DeleteTablet for tablet cef85debdd9b4023aaeb9f369674eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:33 UTC) from {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:35928
I20260926 09:22:33.293855 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 32350
I20260926 09:22:33.294811 32639 tablet_service.cc:1514] Processing DeleteTablet for tablet cef85debdd9b4023aaeb9f369674eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:33 UTC) from {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:56316
I20260926 09:22:33.295073 306 tablet_service.cc:1514] Processing DeleteTablet for tablet cef85debdd9b4023aaeb9f369674eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:33 UTC) from {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:39686
I20260926 09:22:33.295117 307 tablet_service.cc:1514] Processing DeleteTablet for tablet cef85debdd9b4023aaeb9f369674eac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:33 UTC) from {username='oryx', principal='oryx/127.24.235.124@KRBTEST.COM'} at 127.0.0.1:39686
I20260926 09:22:33.295281 487 tablet_replica.cc:330] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: stopping tablet replica
I20260926 09:22:33.295361 487 raft_consensus.cc:2229] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:33.295449 487 raft_consensus.cc:2258] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:33.295745 487 ts_tablet_manager.cc:1905] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:33.297150 486 tablet_replica.cc:330] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: stopping tablet replica
I20260926 09:22:33.297223 486 raft_consensus.cc:2229] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:33.297279 486 raft_consensus.cc:2258] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:33.297389 487 ts_tablet_manager.cc:1918] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:33.297443 487 log.cc:1196] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/cef85debdd9b4023aaeb9f369674eac1
I20260926 09:22:33.297696 487 ts_tablet_manager.cc:1939] T cef85debdd9b4023aaeb9f369674eac1 P 08b113f19e954379945e43267c02b20e: Deleting consensus metadata
I20260926 09:22:33.298056 32373 catalog_manager.cc:4841] TS 08b113f19e954379945e43267c02b20e (127.24.235.67:43077): tablet cef85debdd9b4023aaeb9f369674eac1 (table test-table [id=82de44359d8f4195bea574acdb5cd4f5]) successfully deleted
W20260926 09:22:33.298255 32373 catalog_manager.cc:4808] TS 08b113f19e954379945e43267c02b20e (127.24.235.67:43077): delete failed for tablet cef85debdd9b4023aaeb9f369674eac1 because the tablet was not found. No further retry: Not found: Tablet not found: cef85debdd9b4023aaeb9f369674eac1
I20260926 09:22:33.298372 486 ts_tablet_manager.cc:1905] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:33.299948 486 ts_tablet_manager.cc:1918] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:33.300015 486 log.cc:1196] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/cef85debdd9b4023aaeb9f369674eac1
I20260926 09:22:33.300273 486 ts_tablet_manager.cc:1939] T cef85debdd9b4023aaeb9f369674eac1 P 83ca21e2ed2140ca850897b44cb77d85: Deleting consensus metadata
I20260926 09:22:33.300743 32374 catalog_manager.cc:4841] TS 83ca21e2ed2140ca850897b44cb77d85 (127.24.235.66:35751): tablet cef85debdd9b4023aaeb9f369674eac1 (table test-table [id=82de44359d8f4195bea574acdb5cd4f5]) successfully deleted
W20260926 09:22:33.302860 438 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:33.302576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:40347 (local address 127.0.0.1:35934)
0926 09:22:33.302621 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:33.302633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:33.302673 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:33.302821 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:40347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 09:22:33.303007 32373 catalog_manager.cc:4584] TS 0a8b026bd0a746f09ab0f03faf2f7b0d (127.24.235.65:40347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cef85debdd9b4023aaeb9f369674eac1: Network error: Client connection negotiation failed: client connection to 127.24.235.65:40347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:22:33.303117 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 32576
I20260926 09:22:33.312754 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 32712
I20260926 09:22:33.319054 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 32070
I20260926 09:22:33.324325 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 32257
2026-09-26T09:22:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4534 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:33 dist-test-slave-7zlw krb5kdc[490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:33 dist-test-slave-7zlw krb5kdc[490](info): set up 2 sockets
Sep 26 09:22:33 dist-test-slave-7zlw krb5kdc[490](info): commencing operation
krb5kdc: starting...
W20260926 09:22:34.389833 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:34 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab.
Entry for principal oryx/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab.
I20260926 09:22:34.524677 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46737
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab
--principal=oryx/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:34.608141 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:34.608443 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:34.608480 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:34.609752 503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:34.609828 503 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 09:22:34.609849 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:34.609872 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:34.609903 503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:34.609927 503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:34.611496 503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:46737
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.126.keytab
--principal=oryx/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:34.611784 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:34.612030 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:34.617146 508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:34.617182 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:34.617161 509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:34.617422 503 server_base.cc:1034] running on GCE node
I20260926 09:22:34.617718 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:34.618021 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:34.619217 503 hybrid_clock.cc:648] HybridClock initialized: now 1790414554619208 us; error 53 us; skew 500 ppm
Sep 26 09:22:34 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:34.622143 503 init.cc:377] Logged in from keytab as oryx/127.24.235.126@KRBTEST.COM (short username oryx)
I20260926 09:22:34.622622 503 webserver.cc:460] Webserver started at http://127.24.235.126:39005/ using document root <none> and password file <none>
I20260926 09:22:34.622869 503 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:34.622939 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:34.623088 503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:34.624213 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "be8b40bd64344f4a9cd7cb463a0f199a"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "8b102d698b555c5725afd603d91a54d7"
server_key_iv: "c5ac20e9bdcda4330017ae01a265431a"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.624588 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "be8b40bd64344f4a9cd7cb463a0f199a"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "8b102d698b555c5725afd603d91a54d7"
server_key_iv: "c5ac20e9bdcda4330017ae01a265431a"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.626035 503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:34.626839 518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:34.627017 503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:34.627125 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "be8b40bd64344f4a9cd7cb463a0f199a"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "8b102d698b555c5725afd603d91a54d7"
server_key_iv: "c5ac20e9bdcda4330017ae01a265431a"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.627202 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:34.641427 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:34.643874 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:34.644054 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:34.648928 503 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:46737
I20260926 09:22:34.649062 580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:46737 every 8 connection(s)
I20260926 09:22:34.649389 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:34.650075 581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:34.650511 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 503
I20260926 09:22:34.650632 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:34.650821 25517 external_mini_cluster.cc:1428] Setting key a13a0743a17f767d0f85fc29f3307efd
I20260926 09:22:34.653128 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Bootstrap starting.
I20260926 09:22:34.653770 581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:34.654064 581 log.cc:824] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:34.654834 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: No bootstrap required, opened a new log
I20260926 09:22:34.657502 581 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } }
I20260926 09:22:34.657646 581 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:34.657670 581 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8b40bd64344f4a9cd7cb463a0f199a, State: Initialized, Role: FOLLOWER
I20260926 09:22:34.657804 581 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } }
I20260926 09:22:34.657884 581 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:34.657907 581 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:34.657974 581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 0 FOLLOWER]: Advancing to term 1
Sep 26 09:22:34 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:34.658563 581 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } }
I20260926 09:22:34.658650 581 leader_election.cc:302] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8b40bd64344f4a9cd7cb463a0f199a; no voters:
I20260926 09:22:34.658846 581 leader_election.cc:288] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:34.658911 586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:34.659150 581 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:34.659090 586 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [term 1 LEADER]: Becoming Leader. State: Replica: be8b40bd64344f4a9cd7cb463a0f199a, State: Running, Role: LEADER
I20260926 09:22:34.659549 586 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } }
I20260926 09:22:34.661037 587 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8b40bd64344f4a9cd7cb463a0f199a. Latest consensus state: current_term: 1 leader_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } } }
I20260926 09:22:34.661367 587 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:34.661257 588 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8b40bd64344f4a9cd7cb463a0f199a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46737 } } }
W20260926 09:22:34.661389 600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 09:22:34.661476 588 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a [sys.catalog]: This master's current role is: LEADER
W20260926 09:22:34.661489 600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:34.661662 603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:34.661854 603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:34.662353 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:34.651407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.24.235.126:46737)
0926 09:22:34.651581 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:34.651591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:34.651665 (+ 74us) server_negotiation.cc:407] Connection header received
0926 09:22:34.652071 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:34.652077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:34.652125 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:34.652253 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:34.653121 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.654192 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.655178 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.655572 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.655662 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:34.658517 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:34.658521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:34.658522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:34.658534 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:34.660418 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.660842 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.660845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.660845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.660972 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.661366 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.661367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.661367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.661740 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:34.661899 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:34.662003 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 09:22:34.662159 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260926 09:22:34.665505 603 catalog_manager.cc:1347] Generated new cluster ID: de60db7d8be14a2bb67cd12446fc2c71
I20260926 09:22:34.665575 603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:34.689550 603 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:34.690164 603 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:34.698999 603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Generated new TSK 0
I20260926 09:22:34.699311 603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
I20260926 09:22:34.740732 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46737
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:34.839877 607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:34.840123 607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:34.840152 607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:34.841370 607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:34.841416 607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:34.841439 607 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:34.841461 607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:34.843039 607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:34.843254 607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:34.843446 607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:34.849213 613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:34.849257 612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:34.849339 607 server_base.cc:1034] running on GCE node
W20260926 09:22:34.849231 615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:34.849761 607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:34.850063 607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:34.851246 607 hybrid_clock.cc:648] HybridClock initialized: now 1790414554851223 us; error 35 us; skew 500 ppm
Sep 26 09:22:34 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:34.854233 607 init.cc:377] Logged in from keytab as oryx/127.24.235.65@KRBTEST.COM (short username oryx)
I20260926 09:22:34.854722 607 webserver.cc:460] Webserver started at http://127.24.235.65:39763/ using document root <none> and password file <none>
I20260926 09:22:34.854977 607 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:34.855028 607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:34.855177 607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:34.856372 607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "c7c9bf89bfc49d238d58202a919af14d"
server_key_iv: "ccb85b6f38a304020ff751ba6831720f"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.856801 607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "c7c9bf89bfc49d238d58202a919af14d"
server_key_iv: "ccb85b6f38a304020ff751ba6831720f"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.858408 607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:34.859750 622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:34.859930 607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:34.860031 607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe"
format_stamp: "Formatted at 2026-09-26 09:22:34 on dist-test-slave-7zlw"
server_key: "c7c9bf89bfc49d238d58202a919af14d"
server_key_iv: "ccb85b6f38a304020ff751ba6831720f"
server_key_version: "encryptionkey@0"
I20260926 09:22:34.860142 607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:34.868983 607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:34.871738 607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:34.871874 607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:34.872406 607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:34.872453 607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:34.872485 607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:34.872550 607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:34.880568 607 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46775
I20260926 09:22:34.880738 737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46775 every 8 connection(s)
I20260926 09:22:34.881160 607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:22:34 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:34.885106 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:34.872990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:46247 (local address 127.24.235.126:46737)
0926 09:22:34.873090 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:34.873092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:34.873351 (+ 259us) server_negotiation.cc:407] Connection header received
0926 09:22:34.874232 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:34.874233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:34.874247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:34.874331 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:34.875527 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.876421 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.878332 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.878726 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.878815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:34.882030 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:34.882031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:34.882032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:34.882051 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:34.883489 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.883970 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.883971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.883972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.884032 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.884336 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.884337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.884338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.884478 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:34.884543 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:34.884874 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 09:22:34.884959 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 09:22:34.885269 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:34.872955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.65:46247)
0926 09:22:34.873170 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:34.873232 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:34.874051 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:34.874438 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:34.874444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:34.874900 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:34.875273 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:34.875280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.876561 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.876562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:34.878210 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:34.878214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.878840 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.878841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:34.878955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:34.879754 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:34.879765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:34.881913 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:34.883576 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:34.883583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:34.883584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:34.883875 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:34.884119 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:34.884120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:34.884120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:34.884222 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:34.884561 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:34.884567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:34.884735 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 09:22:34.884821 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:22:34.884951 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":119,"threads_started":1}
I20260926 09:22:34.885901 738 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46737
I20260926 09:22:34.885998 738 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:34.886211 738 heartbeater.cc:511] Master 127.24.235.126:46737 requested a full tablet report, sending...
I20260926 09:22:34.886904 534 ts_manager.cc:194] Registered new tserver with Master: 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775)
I20260926 09:22:34.887414 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 607
I20260926 09:22:34.887542 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:34.887825 25517 external_mini_cluster.cc:1428] Setting key ede395a395eeb709a7720a00bbb0db67
I20260926 09:22:34.887827 534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:46247
I20260926 09:22:34.894455 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:34.887148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55431 (local address 127.24.235.126:46737)
0926 09:22:34.887191 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:34.887192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:34.887400 (+ 208us) server_negotiation.cc:407] Connection header received
0926 09:22:34.887432 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:34.887433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:34.887446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:34.887517 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:34.888006 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.888764 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.890055 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.890373 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.890456 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:34.891571 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:34.891573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:34.891573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:34.891580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:34.893140 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.893610 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.893611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.893612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.893673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:34.893937 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:34.893938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:34.893938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:34.894046 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:34.894119 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:34.894324 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 09:22:34.894389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:34.894629 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:34.887094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.65:55431)
0926 09:22:34.887311 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:34.887323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:34.887387 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:34.887573 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:34.887575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:34.887805 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:34.887893 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:34.887896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.888899 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.888900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:34.889956 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:34.889960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:34.890470 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:34.890471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:34.890524 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:34.891036 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:34.891041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:34.891473 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:34.893226 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:34.893227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:34.893228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:34.893505 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:34.893772 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:34.893773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:34.893773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:34.893862 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:34.894119 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:34.894121 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:34.894210 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 09:22:34.894309 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 09:22:34.894450 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":161,"threads_started":1}
WARNING: no policy specified for oryx/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
I20260926 09:22:34.916806 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46737
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:35.006490 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:35.006834 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:35.006881 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:35.008826 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:35.008917 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:35.008955 747 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:35.008997 747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:35.010744 747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:35.011054 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:35.011327 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:35.016738 753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:35.016799 755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:35.016764 752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:35.016999 747 server_base.cc:1034] running on GCE node
I20260926 09:22:35.017297 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:35.017591 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:35.018781 747 hybrid_clock.cc:648] HybridClock initialized: now 1790414555018756 us; error 30 us; skew 500 ppm
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:35.021425 747 init.cc:377] Logged in from keytab as oryx/127.24.235.66@KRBTEST.COM (short username oryx)
I20260926 09:22:35.021945 747 webserver.cc:460] Webserver started at http://127.24.235.66:32947/ using document root <none> and password file <none>
I20260926 09:22:35.022212 747 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:35.022262 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:35.022356 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:35.023416 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "956949854f23eaad1235e4e7328952ee"
server_key_iv: "ad5ea8030ef5c3a39be7cced3d2a05d0"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.023736 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "956949854f23eaad1235e4e7328952ee"
server_key_iv: "ad5ea8030ef5c3a39be7cced3d2a05d0"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.025085 747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:35.026078 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.026386 747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.026503 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "956949854f23eaad1235e4e7328952ee"
server_key_iv: "ad5ea8030ef5c3a39be7cced3d2a05d0"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.026640 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:35.034307 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:35.036274 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:35.036437 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:35.037135 747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:35.037191 747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.037235 747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:35.037267 747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.043968 747 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:37425
I20260926 09:22:35.044044 877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:37425 every 8 connection(s)
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:35.044390 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:35.048310 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.037545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:33257 (local address 127.24.235.126:46737)
0926 09:22:35.037609 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.037612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.037826 (+ 214us) server_negotiation.cc:407] Connection header received
0926 09:22:35.038309 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.038311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.038324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.038410 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.039275 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.040109 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.041801 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.042137 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.042228 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.045094 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.045096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.045096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.045106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.046707 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.047165 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.047167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.047167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.047230 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.047604 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.047606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.047606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.047737 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.047817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.048144 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.048240 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:22:35.048482 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.037517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.66:33257)
0926 09:22:35.037693 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.037737 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.038213 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.038460 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.038463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.038859 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:35.039163 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.039170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.040252 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.040253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.041697 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.041701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.042236 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.042237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.042344 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.042931 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:35.042942 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:35.044871 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:35.046804 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.046811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.046813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.047039 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.047327 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.047328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.047328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.047449 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.047857 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:35.047863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:35.047991 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.048123 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.048266 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260926 09:22:35.049127 878 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46737
I20260926 09:22:35.049217 878 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:35.049407 878 heartbeater.cc:511] Master 127.24.235.126:46737 requested a full tablet report, sending...
I20260926 09:22:35.049885 534 ts_manager.cc:194] Registered new tserver with Master: 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425)
I20260926 09:22:35.050423 534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:33257
I20260926 09:22:35.052663 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 747
I20260926 09:22:35.052819 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:35.053079 25517 external_mini_cluster.cc:1428] Setting key bf4363af6509c087381fcecd18a378c4
I20260926 09:22:35.056087 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.050385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:46145 (local address 127.24.235.126:46737)
0926 09:22:35.050430 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.050432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.050580 (+ 148us) server_negotiation.cc:407] Connection header received
0926 09:22:35.050604 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.050604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.050614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.050663 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.050993 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.051465 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.052170 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.052370 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.052454 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.053549 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.053551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.053551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.053557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.054812 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.055244 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.055245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.055246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.055321 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.055548 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.055549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.055549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.055687 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.055742 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.055955 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.056025 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:35.056155 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.050322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.66:46145)
0926 09:22:35.050486 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.050497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.050552 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.050666 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.050667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.050852 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:35.050917 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.050918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.051567 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.051568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.052087 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.052089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.052479 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.052480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.052541 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.053111 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:35.053116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:35.053443 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:35.054898 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.054899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.054900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.055141 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.055404 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.055405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.055405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.055480 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.055754 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:35.055755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:35.055818 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.055898 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.056017 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
I20260926 09:22:35.083621 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46737
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:35.169220 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:35.169425 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:35.169456 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:35.170569 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:35.170606 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:35.170627 887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 09:22:35.170645 887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:35.172106 887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.235.126:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:35.172317 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:35.172519 887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:35.178515 892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:35.178643 893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:35.178642 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:35.178797 887 server_base.cc:1034] running on GCE node
I20260926 09:22:35.179091 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:35.179387 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:35.180590 887 hybrid_clock.cc:648] HybridClock initialized: now 1790414555180565 us; error 44 us; skew 500 ppm
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:35.183357 887 init.cc:377] Logged in from keytab as oryx/127.24.235.67@KRBTEST.COM (short username oryx)
I20260926 09:22:35.183838 887 webserver.cc:460] Webserver started at http://127.24.235.67:43803/ using document root <none> and password file <none>
I20260926 09:22:35.184121 887 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:35.184202 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:35.184367 887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:35.185633 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "bda94b25fd82420ea7a720bbb05deece"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "babc5835e6e232de04b961ff6efe4706"
server_key_iv: "6b820692eb219b84259cd10283a4c135"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.185997 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "bda94b25fd82420ea7a720bbb05deece"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "babc5835e6e232de04b961ff6efe4706"
server_key_iv: "6b820692eb219b84259cd10283a4c135"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.187587 887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:35.188539 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.188774 887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.188881 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "bda94b25fd82420ea7a720bbb05deece"
format_stamp: "Formatted at 2026-09-26 09:22:35 on dist-test-slave-7zlw"
server_key: "babc5835e6e232de04b961ff6efe4706"
server_key_iv: "6b820692eb219b84259cd10283a4c135"
server_key_version: "encryptionkey@0"
I20260926 09:22:35.188977 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:35.209187 887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:35.211665 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:35.211862 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:35.212486 887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:35.212563 887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.212604 887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:35.212653 887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:35.221091 887 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:34839
I20260926 09:22:35.221182 1017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:34839 every 8 connection(s)
I20260926 09:22:35.221583 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:35.223613 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.213168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:33461 (local address 127.24.235.126:46737)
0926 09:22:35.213216 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.213218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.213370 (+ 152us) server_negotiation.cc:407] Connection header received
0926 09:22:35.213928 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.213930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.213942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.213990 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.214781 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.215315 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.216421 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.216640 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.216723 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.220378 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.220380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.220381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.220390 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.222293 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.222718 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.222719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.222719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.222780 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.223006 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.223007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.223007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.223119 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.223174 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.223416 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.223540 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:22:35.223703 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.213038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.67:33461)
0926 09:22:35.213221 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.213270 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.213824 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.214131 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.214135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.214392 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:35.214669 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.214674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.215436 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.215437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.216323 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.216326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.216730 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.216731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.216890 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.217580 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:35.217593 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:35.220262 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:35.222382 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.222386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.222388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.222611 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.222847 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.222848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.222848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.222948 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.223178 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:35.223182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:35.223304 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.223389 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.223536 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260926 09:22:35.224200 1018 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46737
I20260926 09:22:35.224287 1018 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:35.224478 1018 heartbeater.cc:511] Master 127.24.235.126:46737 requested a full tablet report, sending...
I20260926 09:22:35.224990 535 ts_manager.cc:194] Registered new tserver with Master: bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.225481 535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:33461
I20260926 09:22:35.229970 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 887
I20260926 09:22:35.230080 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:35.230300 25517 external_mini_cluster.cc:1428] Setting key 9096721fccc818f42e934bd544d46d2c
I20260926 09:22:35.231410 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:35.231640 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.225470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:38543 (local address 127.24.235.126:46737)
0926 09:22:35.225514 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.225516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.225627 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:22:35.225697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.225697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.225707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.225754 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.226083 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.226622 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.227307 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.227479 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.227534 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.228453 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.228455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.228455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.228461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.230265 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.230738 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.230739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.230740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.230803 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.231017 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.231018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.231018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.231174 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.231247 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.231460 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.231560 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:35.231640 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.225381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.24.235.67:38543)
0926 09:22:35.225563 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.225575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.225649 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.225767 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.225768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.225951 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:35.226009 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.226011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.226731 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.226731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.227234 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.227236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.227561 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.227562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.227612 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.228060 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:35.228065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:35.228369 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:35.230371 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.230372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.230373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.230669 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.230872 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.230873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.230874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.230952 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.231281 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:35.231282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:35.231340 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.231417 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.231531 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:22:35.254621 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:35.248479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.24.235.126:46737)
0926 09:22:35.248614 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.248616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.248628 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:35.248660 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.248661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.248676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.248741 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.249554 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.250044 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.250860 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.251068 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.251128 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.254536 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59140: BlockingRecv error: recv got EOF from 127.0.0.1:59140 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:35.263340 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.255547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.24.235.126:46737)
0926 09:22:35.255672 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.255675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.255693 (+ 18us) server_negotiation.cc:407] Connection header received
0926 09:22:35.255716 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.255716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.255725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.255785 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.256219 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.256748 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.257575 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.257775 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.257844 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.260370 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.260372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.260372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.260379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.262117 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.262447 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.262448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.262448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.262505 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.262717 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.262717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.262718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.262830 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.262914 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.263103 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.263242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 09:22:35.265221 535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:35.266844 535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:35.273934 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.24.235.65:46775)
0926 09:22:35.269726 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.269733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.269783 (+ 50us) server_negotiation.cc:407] Connection header received
0926 09:22:35.269915 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.269917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.269935 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.269993 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.270299 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271468 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.272652 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273453 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273540 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.273638 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.273773 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.273820 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260926 09:22:35.274003 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37425 (local address 127.0.0.1:50872)
0926 09:22:35.269524 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.269570 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.269711 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.270101 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.270103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.270105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.270211 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.270213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271546 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.271546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.272767 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.272769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273646 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.273719 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.273723 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.273757 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.273823 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":34,"thread_start_us":49,"threads_started":1}
I20260926 09:22:35.274247 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.24.235.66:37425)
0926 09:22:35.269932 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.269936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.269945 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:35.269987 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.269989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.270003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.270290 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.270351 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271447 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.272870 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273531 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273963 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.274065 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.274099 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.274155 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":108,"threads_started":1}
I20260926 09:22:35.274284 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.24.235.67:34839)
0926 09:22:35.269983 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.269988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.270000 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:35.270042 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.270044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.270061 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.270144 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.270373 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271116 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.272795 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273519 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273576 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.273647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.274125 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.274183 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260926 09:22:35.273996 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34839 (local address 127.0.0.1:42642)
0926 09:22:35.269767 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.269777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.269818 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.270214 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.270215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.270217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.270288 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.270289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271226 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.271227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.272701 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.272704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273681 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.273781 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.273786 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.273838 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.273880 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260926 09:22:35.274873 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.269373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46775 (local address 127.0.0.1:37520)
0926 09:22:35.269708 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.269719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.269812 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.270100 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.270103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.270108 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.270205 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.270207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.271583 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.271584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.272560 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.272562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.273552 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.273554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.273650 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.273666 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.273747 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.273823 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":45,"thread_start_us":89,"threads_started":1}
I20260926 09:22:35.275000 812 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb2620af90455aa448c27bc4ba155d (DEFAULT_TABLE table=test-table [id=973c8c1515e5450aaafbeeac1afd56f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.275070 952 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb2620af90455aa448c27bc4ba155d (DEFAULT_TABLE table=test-table [id=973c8c1515e5450aaafbeeac1afd56f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.275439 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04eb2620af90455aa448c27bc4ba155d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.275439 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04eb2620af90455aa448c27bc4ba155d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.278550 1046 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Bootstrap starting.
I20260926 09:22:35.278915 1047 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Bootstrap starting.
I20260926 09:22:35.279259 1046 tablet_bootstrap.cc:654] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.279284 672 tablet_service.cc:1467] Processing CreateTablet for tablet 04eb2620af90455aa448c27bc4ba155d (DEFAULT_TABLE table=test-table [id=973c8c1515e5450aaafbeeac1afd56f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.279608 1046 log.cc:824] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:35.279637 1047 tablet_bootstrap.cc:654] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.279721 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04eb2620af90455aa448c27bc4ba155d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.279963 1047 log.cc:824] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:35.280478 1046 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: No bootstrap required, opened a new log
I20260926 09:22:35.280551 1046 ts_tablet_manager.cc:1397] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:35.280740 1047 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: No bootstrap required, opened a new log
I20260926 09:22:35.280800 1047 ts_tablet_manager.cc:1397] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:35.282670 1046 raft_consensus.cc:348] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.282858 1046 raft_consensus.cc:374] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.282894 1046 raft_consensus.cc:729] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda94b25fd82420ea7a720bbb05deece, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.282807 1047 raft_consensus.cc:348] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.282935 1047 raft_consensus.cc:374] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.282964 1047 raft_consensus.cc:729] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.283025 1046 consensus_queue.cc:260] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.283083 1047 consensus_queue.cc:260] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.283324 1047 ts_tablet_manager.cc:1428] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:22:35.283324 1046 ts_tablet_manager.cc:1428] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:22:35.283358 1018 heartbeater.cc:503] Master 127.24.235.126:46737 was elected leader, sending a full tablet report...
I20260926 09:22:35.283455 878 heartbeater.cc:503] Master 127.24.235.126:46737 was elected leader, sending a full tablet report...
I20260926 09:22:35.283752 1052 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Bootstrap starting.
I20260926 09:22:35.284426 1052 tablet_bootstrap.cc:654] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.284832 1052 log.cc:824] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:35.285742 1052 tablet_bootstrap.cc:492] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: No bootstrap required, opened a new log
I20260926 09:22:35.285821 1052 ts_tablet_manager.cc:1397] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:22:35.287268 1052 raft_consensus.cc:348] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.287405 1052 raft_consensus.cc:374] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.287425 1052 raft_consensus.cc:729] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.287559 1052 consensus_queue.cc:260] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.287822 1052 ts_tablet_manager.cc:1428] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:22:35.287940 738 heartbeater.cc:503] Master 127.24.235.126:46737 was elected leader, sending a full tablet report...
W20260926 09:22:35.295004 879 tablet.cc:2369] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:22:35.381999 739 tablet.cc:2369] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:35.403636 1051 raft_consensus.cc:482] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:35.403765 1051 raft_consensus.cc:504] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.404059 1051 leader_election.cc:288] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.407171 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.404329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46775 (local address 127.24.235.66:37399)
0926 09:22:35.404397 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.404410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.404461 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.404718 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.404719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.404720 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.404790 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.404792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.405506 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.405507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.406286 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.406288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.406863 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.406959 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.406961 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.407034 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.407076 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 09:22:35.407158 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.404405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:37399 (local address 127.24.235.65:46775)
0926 09:22:35.404479 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.404482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.404493 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:35.404645 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.404646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.404657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.404702 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.404870 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.405406 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406382 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.406778 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406866 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.406875 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.407039 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.407075 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:22:35.407258 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.404684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:55349 (local address 127.24.235.67:34839)
0926 09:22:35.404743 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.404745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.404783 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:22:35.404895 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.404896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.404905 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.404962 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.405126 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.405581 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406489 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.406902 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406992 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.406999 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.407160 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.407192 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:22:35.407270 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.404470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34839 (local address 127.24.235.66:55349)
0926 09:22:35.404713 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.404723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.404812 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.404974 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.404975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.404976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.405069 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.405071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.405670 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.405671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.406391 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.406393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.406993 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.406994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.407066 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.407075 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.407153 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.407190 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":173,"threads_started":1}
I20260926 09:22:35.407590 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04eb2620af90455aa448c27bc4ba155d" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" is_pre_election: true
I20260926 09:22:35.407590 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04eb2620af90455aa448c27bc4ba155d" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece" is_pre_election: true
I20260926 09:22:35.407763 692 raft_consensus.cc:2454] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 0.
I20260926 09:22:35.407773 972 raft_consensus.cc:2454] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 0.
I20260926 09:22:35.408028 766 leader_election.cc:302] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:35.408179 1051 raft_consensus.cc:2781] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:35.408267 1051 raft_consensus.cc:482] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:35.408285 1051 raft_consensus.cc:3037] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.409298 1051 raft_consensus.cc:504] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.409442 1051 leader_election.cc:288] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 election: Requested vote from peers 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.409652 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04eb2620af90455aa448c27bc4ba155d" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe"
I20260926 09:22:35.409701 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04eb2620af90455aa448c27bc4ba155d" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece"
I20260926 09:22:35.409761 692 raft_consensus.cc:3037] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.409758 972 raft_consensus.cc:3037] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.410442 972 raft_consensus.cc:2454] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 1.
I20260926 09:22:35.410593 692 raft_consensus.cc:2454] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 1.
I20260926 09:22:35.410615 766 leader_election.cc:302] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88c8ae85299f4a98843fe71b2ed3fdc8, bda94b25fd82420ea7a720bbb05deece; no voters:
I20260926 09:22:35.410754 1051 raft_consensus.cc:2781] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:35.410897 1051 raft_consensus.cc:686] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 LEADER]: Becoming Leader. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Running, Role: LEADER
I20260926 09:22:35.410996 1051 consensus_queue.cc:237] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.411859 535 catalog_manager.cc:5495] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:35.418947 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.416413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.24.235.66:37425)
0926 09:22:35.416487 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.416490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.416509 (+ 19us) server_negotiation.cc:407] Connection header received
0926 09:22:35.416545 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.416546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.416564 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.416638 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.416757 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.417344 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.418224 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.418423 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.418484 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.418678 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.418776 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.418837 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.418870 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.418890 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:22:35.420782 972 raft_consensus.cc:1261] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Refusing update from remote peer 88c8ae85299f4a98843fe71b2ed3fdc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.420783 692 raft_consensus.cc:1261] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Refusing update from remote peer 88c8ae85299f4a98843fe71b2ed3fdc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.421150 1051 consensus_queue.cc:1035] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.421296 1056 consensus_queue.cc:1035] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.423607 1059 mvcc.cc:205] Tried to move back new op lower bound from 7333538019000750080 to 7333538018964066304. Current Snapshot: MvccSnapshot[applied={T|T < 7333538019000750080}]
I20260926 09:22:35.424880 1061 mvcc.cc:205] Tried to move back new op lower bound from 7333538019000750080 to 7333538018964066304. Current Snapshot: MvccSnapshot[applied={T|T < 7333538019000750080}]
I20260926 09:22:35.425048 1060 mvcc.cc:205] Tried to move back new op lower bound from 7333538019000750080 to 7333538018964066304. Current Snapshot: MvccSnapshot[applied={T|T < 7333538019000750080}]
I20260926 09:22:35.426394 534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.426505 534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.427397 534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Sending DeleteTablet for 3 replicas of tablet 04eb2620af90455aa448c27bc4ba155d
I20260926 09:22:35.427754 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 04eb2620af90455aa448c27bc4ba155d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260926 09:22:35.427855 952 tablet_service.cc:1514] Processing DeleteTablet for tablet 04eb2620af90455aa448c27bc4ba155d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 09:22:35.428020 672 tablet_service.cc:1514] Processing DeleteTablet for tablet 04eb2620af90455aa448c27bc4ba155d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:37520
I20260926 09:22:35.427987 534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:35.428133 534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:35.428246 1070 tablet_replica.cc:330] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: stopping tablet replica
I20260926 09:22:35.428354 1070 raft_consensus.cc:2229] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.428356 1069 tablet_replica.cc:330] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: stopping tablet replica
I20260926 09:22:35.428473 1070 raft_consensus.cc:2258] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.428601 1069 raft_consensus.cc:2229] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.428990 1070 ts_tablet_manager.cc:1905] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.429055 1069 raft_consensus.cc:2258] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.429606 672 tablet_service.cc:1467] Processing CreateTablet for tablet 6724d7834ca2461292f14e01246839ec (DEFAULT_TABLE table=test-table [id=ebe8e13fb73b46738f63d2f1f437985c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.429656 1069 ts_tablet_manager.cc:1905] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.429741 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6724d7834ca2461292f14e01246839ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.431164 1052 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Bootstrap starting.
I20260926 09:22:35.431221 952 tablet_service.cc:1467] Processing CreateTablet for tablet 6724d7834ca2461292f14e01246839ec (DEFAULT_TABLE table=test-table [id=ebe8e13fb73b46738f63d2f1f437985c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.431339 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6724d7834ca2461292f14e01246839ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.431581 1052 tablet_bootstrap.cc:654] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.431739 1069 ts_tablet_manager.cc:1918] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.431849 1069 log.cc:1196] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/04eb2620af90455aa448c27bc4ba155d
I20260926 09:22:35.431936 1070 ts_tablet_manager.cc:1918] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.432013 1070 log.cc:1196] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/04eb2620af90455aa448c27bc4ba155d
I20260926 09:22:35.432327 1069 ts_tablet_manager.cc:1939] T 04eb2620af90455aa448c27bc4ba155d P bda94b25fd82420ea7a720bbb05deece: Deleting consensus metadata
I20260926 09:22:35.432330 1070 ts_tablet_manager.cc:1939] T 04eb2620af90455aa448c27bc4ba155d P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting consensus metadata
I20260926 09:22:35.432698 812 tablet_service.cc:1467] Processing CreateTablet for tablet 6724d7834ca2461292f14e01246839ec (DEFAULT_TABLE table=test-table [id=ebe8e13fb73b46738f63d2f1f437985c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.432819 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6724d7834ca2461292f14e01246839ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.433000 1052 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: No bootstrap required, opened a new log
I20260926 09:22:35.433073 1052 ts_tablet_manager.cc:1397] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:22:35.433440 1052 raft_consensus.cc:348] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.433521 1052 raft_consensus.cc:374] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.433542 1052 raft_consensus.cc:729] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.433571 1052 consensus_queue.cc:260] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.433702 1052 ts_tablet_manager.cc:1428] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:35.434219 1047 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Bootstrap starting.
I20260926 09:22:35.434249 1046 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Bootstrap starting.
I20260926 09:22:35.434659 1047 tablet_bootstrap.cc:654] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.434664 1046 tablet_bootstrap.cc:654] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.435290 522 catalog_manager.cc:4841] TS bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839): tablet 04eb2620af90455aa448c27bc4ba155d (table test-table [id=973c8c1515e5450aaafbeeac1afd56f4]) successfully deleted
I20260926 09:22:35.435353 1046 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: No bootstrap required, opened a new log
I20260926 09:22:35.435389 1046 ts_tablet_manager.cc:1397] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.435405 522 catalog_manager.cc:4841] TS 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775): tablet 04eb2620af90455aa448c27bc4ba155d (table test-table [id=973c8c1515e5450aaafbeeac1afd56f4]) successfully deleted
I20260926 09:22:35.435545 1046 raft_consensus.cc:348] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.435618 1046 raft_consensus.cc:374] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.435637 1046 raft_consensus.cc:729] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda94b25fd82420ea7a720bbb05deece, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.435686 1046 consensus_queue.cc:260] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.435760 1046 ts_tablet_manager.cc:1428] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.436127 1047 tablet_bootstrap.cc:492] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: No bootstrap required, opened a new log
I20260926 09:22:35.436178 1047 ts_tablet_manager.cc:1397] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:35.436324 1047 raft_consensus.cc:348] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.436399 1047 raft_consensus.cc:374] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.436429 1047 raft_consensus.cc:729] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.436475 1047 consensus_queue.cc:260] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.436575 1047 ts_tablet_manager.cc:1428] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.441177 1068 tablet_replica.cc:330] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: stopping tablet replica
I20260926 09:22:35.441330 1068 raft_consensus.cc:2229] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:35.441448 1068 raft_consensus.cc:2258] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.441941 1068 ts_tablet_manager.cc:1905] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.443840 1068 ts_tablet_manager.cc:1918] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.443917 1068 log.cc:1196] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/04eb2620af90455aa448c27bc4ba155d
I20260926 09:22:35.444217 1068 ts_tablet_manager.cc:1939] T 04eb2620af90455aa448c27bc4ba155d P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting consensus metadata
I20260926 09:22:35.444834 521 catalog_manager.cc:4841] TS 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425): tablet 04eb2620af90455aa448c27bc4ba155d (table test-table [id=973c8c1515e5450aaafbeeac1afd56f4]) successfully deleted
W20260926 09:22:35.472215 1019 tablet.cc:2369] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:35.560645 1054 raft_consensus.cc:482] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:35.560747 1054 raft_consensus.cc:504] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.561110 1054 leader_election.cc:288] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.565346 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.561318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34839 (local address 127.24.235.65:56135)
0926 09:22:35.561485 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.561495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.561541 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.561708 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.561709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.561710 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.561821 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.561824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.562886 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.562888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.564390 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.564394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.565096 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.565098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.565180 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.565183 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.565233 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.565267 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":116,"thread_start_us":109,"threads_started":1}
I20260926 09:22:35.565376 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.561422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:56135 (local address 127.24.235.67:34839)
0926 09:22:35.561470 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.561473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.561546 (+ 73us) server_negotiation.cc:407] Connection header received
0926 09:22:35.561623 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.561624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.561634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.561692 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.561902 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.562760 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.564565 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.564938 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.565051 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.565062 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.565259 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.565302 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:35.565783 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6724d7834ca2461292f14e01246839ec" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece" is_pre_election: true
I20260926 09:22:35.565909 972 raft_consensus.cc:2454] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:35.566087 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.561316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37425 (local address 127.24.235.65:48593)
0926 09:22:35.561451 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.561466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.561541 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.561706 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.561707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.561710 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:35.561821 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.561824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.562888 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.562888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.564390 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.564394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.565804 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.565806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.565865 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.565869 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.565950 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.565995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":89,"thread_start_us":73,"threads_started":1}
I20260926 09:22:35.566165 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:48593 (local address 127.24.235.66:37425)
0926 09:22:35.561539 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.561542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.561553 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:35.561611 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.561612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.561625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.561692 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:35.561916 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.562796 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.565107 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.565688 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.565760 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.565774 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.566043 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.566090 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:35.566233 626 leader_election.cc:302] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, bda94b25fd82420ea7a720bbb05deece; no voters:
I20260926 09:22:35.566375 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6724d7834ca2461292f14e01246839ec" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" is_pre_election: true
I20260926 09:22:35.566431 1054 raft_consensus.cc:2781] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:35.566530 1054 raft_consensus.cc:482] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:35.566506 832 raft_consensus.cc:2454] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:35.566553 1054 raft_consensus.cc:3037] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.567303 1054 raft_consensus.cc:504] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.567469 1054 leader_election.cc:288] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Requested vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.567657 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6724d7834ca2461292f14e01246839ec" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece"
I20260926 09:22:35.567720 972 raft_consensus.cc:3037] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.567704 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6724d7834ca2461292f14e01246839ec" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
I20260926 09:22:35.567806 832 raft_consensus.cc:3037] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.568374 972 raft_consensus.cc:2454] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:35.568548 832 raft_consensus.cc:2454] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:35.568542 626 leader_election.cc:302] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, bda94b25fd82420ea7a720bbb05deece; no voters:
I20260926 09:22:35.568662 1054 raft_consensus.cc:2781] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:35.568845 1054 raft_consensus.cc:686] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Becoming Leader. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Running, Role: LEADER
I20260926 09:22:35.568938 1054 consensus_queue.cc:237] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.569738 535 catalog_manager.cc:5495] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65). New cstate: current_term: 1 leader_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } health_report { overall_health: UNKNOWN } } }
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414554, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.126@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:35.613323 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.605904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46737 (local address 127.0.0.1:59158)
0926 09:22:35.606102 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.606122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.606198 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.606332 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.606333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.606739 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:35.606794 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.606796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.607495 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.607496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.608045 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.608047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.608439 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.608440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.608539 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.609085 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:35.609089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:35.610608 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:35.612218 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.612225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.612227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.612442 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.612707 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:35.612707 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:35.612708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:35.612769 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:35.612995 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:35.613001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:35.613071 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.613129 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.613225 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260926 09:22:35.613407 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.606050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.24.235.126:46737)
0926 09:22:35.606138 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.606140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.606194 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:22:35.606246 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.606247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.606255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.606307 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.606872 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.607358 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.608152 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.608351 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.608409 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.610701 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.610702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.610703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.610708 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.612128 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.612585 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.612587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.612587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.612638 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.612828 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.612829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.612829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.612938 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.612986 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.613163 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.613271 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":68}
I20260926 09:22:35.615348 535 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:59158:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 09:22:35.615614 535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:35.617450 672 tablet_service.cc:1467] Processing CreateTablet for tablet 71148cfda3644ac39e118cc08b47eff4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8c27b590055445490bca17b243d19ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:35.617448 812 tablet_service.cc:1467] Processing CreateTablet for tablet 71148cfda3644ac39e118cc08b47eff4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8c27b590055445490bca17b243d19ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:35.617615 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71148cfda3644ac39e118cc08b47eff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.617615 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71148cfda3644ac39e118cc08b47eff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.617751 952 tablet_service.cc:1467] Processing CreateTablet for tablet 71148cfda3644ac39e118cc08b47eff4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8c27b590055445490bca17b243d19ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 09:22:35.617892 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71148cfda3644ac39e118cc08b47eff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.619035 1046 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: Bootstrap starting.
I20260926 09:22:35.619112 1047 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: Bootstrap starting.
I20260926 09:22:35.619132 1052 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Bootstrap starting.
I20260926 09:22:35.619493 1047 tablet_bootstrap.cc:654] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.619511 1052 tablet_bootstrap.cc:654] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.619530 1046 tablet_bootstrap.cc:654] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.620237 1047 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: No bootstrap required, opened a new log
I20260926 09:22:35.620337 1047 ts_tablet_manager.cc:1397] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.620368 1052 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: No bootstrap required, opened a new log
I20260926 09:22:35.620417 1052 ts_tablet_manager.cc:1397] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:35.620513 1046 tablet_bootstrap.cc:492] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: No bootstrap required, opened a new log
I20260926 09:22:35.620493 1047 raft_consensus.cc:348] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620551 1046 ts_tablet_manager.cc:1397] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:22:35.620537 1052 raft_consensus.cc:348] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620574 1047 raft_consensus.cc:374] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.620594 1052 raft_consensus.cc:374] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.620608 1052 raft_consensus.cc:729] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.620601 1047 raft_consensus.cc:729] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.620659 1047 consensus_queue.cc:260] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620669 1052 consensus_queue.cc:260] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620755 1047 ts_tablet_manager.cc:1428] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:35.620726 1046 raft_consensus.cc:348] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620788 1052 ts_tablet_manager.cc:1428] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.620795 1046 raft_consensus.cc:374] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.620823 1046 raft_consensus.cc:729] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda94b25fd82420ea7a720bbb05deece, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.620828 1054 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.620873 1046 consensus_queue.cc:260] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.620894 1054 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.620870 1056 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.620942 1056 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.620968 1046 ts_tablet_manager.cc:1428] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.621042 1050 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.621137 1050 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.832216 1056 raft_consensus.cc:482] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:35.832330 1056 raft_consensus.cc:504] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.832587 1056 leader_election.cc:288] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.832937 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71148cfda3644ac39e118cc08b47eff4" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" is_pre_election: true
I20260926 09:22:35.833077 692 raft_consensus.cc:2454] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 0.
I20260926 09:22:35.832933 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71148cfda3644ac39e118cc08b47eff4" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece" is_pre_election: true
I20260926 09:22:35.833179 972 raft_consensus.cc:2454] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 0.
I20260926 09:22:35.833269 766 leader_election.cc:302] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:35.833384 1056 raft_consensus.cc:2781] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:35.833421 1056 raft_consensus.cc:482] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:35.833436 1056 raft_consensus.cc:3037] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.834141 1056 raft_consensus.cc:504] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.834277 1056 leader_election.cc:288] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 election: Requested vote from peers 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.834452 692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71148cfda3644ac39e118cc08b47eff4" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe"
I20260926 09:22:35.834475 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71148cfda3644ac39e118cc08b47eff4" candidate_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece"
I20260926 09:22:35.834556 972 raft_consensus.cc:3037] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.834561 692 raft_consensus.cc:3037] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.835177 692 raft_consensus.cc:2454] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 1.
I20260926 09:22:35.835218 972 raft_consensus.cc:2454] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c8ae85299f4a98843fe71b2ed3fdc8 in term 1.
I20260926 09:22:35.835393 766 leader_election.cc:302] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:35.835515 1056 raft_consensus.cc:2781] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:35.835565 1056 raft_consensus.cc:686] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 LEADER]: Becoming Leader. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Running, Role: LEADER
I20260926 09:22:35.835614 1056 consensus_queue.cc:237] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } }
I20260926 09:22:35.835907 1051 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: TxnStatusTablet state changed. Reason: New leader 88c8ae85299f4a98843fe71b2ed3fdc8. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.835999 1051 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:35.836133 1088 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 09:22:35.836264 1088 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 09:22:35.836381 533 catalog_manager.cc:5495] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } health_report { overall_health: HEALTHY } } }
I20260926 09:22:35.851172 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.848271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.24.235.66:37425)
0926 09:22:35.848316 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.848320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.848334 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:22:35.848403 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.848404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.848413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.848536 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.848635 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.849235 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.850373 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.850629 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.850701 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.850908 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.850940 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.850987 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.851083 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.851104 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:22:35.851207 1083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.848170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37425 (local address 127.0.0.1:50896)
0926 09:22:35.848270 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.848282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.848329 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.848515 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.848516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.848518 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:35.848577 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.848579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.849311 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.849312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.850254 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.850257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.850722 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.850723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.850794 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.850813 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.851010 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.851018 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.851067 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.851113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 09:22:35.852294 972 raft_consensus.cc:1261] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Refusing update from remote peer 88c8ae85299f4a98843fe71b2ed3fdc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.852336 692 raft_consensus.cc:1261] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Refusing update from remote peer 88c8ae85299f4a98843fe71b2ed3fdc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.852587 1056 consensus_queue.cc:1035] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.852746 1056 consensus_queue.cc:1035] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.853257 1050 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: TxnStatusTablet state changed. Reason: New leader 88c8ae85299f4a98843fe71b2ed3fdc8. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.853361 1050 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.853322 1054 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: TxnStatusTablet state changed. Reason: New leader 88c8ae85299f4a98843fe71b2ed3fdc8. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.853415 1054 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.853991 1051 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.854070 1051 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:35.854730 1050 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.854825 1050 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P bda94b25fd82420ea7a720bbb05deece: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.855459 1056 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.855650 1056 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 88c8ae85299f4a98843fe71b2ed3fdc8: This TxnStatusTablet replica's current role is: LEADER
I20260926 09:22:35.855866 1054 tablet_replica.cc:439] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } }
I20260926 09:22:35.855947 1054 tablet_replica.cc:442] T 71148cfda3644ac39e118cc08b47eff4 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 09:22:35.859596 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.856531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.24.235.65:46775)
0926 09:22:35.856596 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.856598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.856606 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:35.856635 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.856636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.856654 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.856717 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.856873 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.857473 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.858613 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.858911 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.858979 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.859223 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.859365 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.859444 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.859499 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.859521 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:22:35.865557 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.862471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:43311 (local address 127.24.235.66:37425)
0926 09:22:35.862560 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.862562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.862822 (+ 260us) server_negotiation.cc:407] Connection header received
0926 09:22:35.862863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.862864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.862876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.862929 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.863089 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.863702 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.864796 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.865004 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.865133 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.865328 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.865357 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.865399 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.865480 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.865500 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:22:35.865622 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.862405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:37425 (local address 127.24.235.65:43311)
0926 09:22:35.862635 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.862648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.862719 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.862949 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.862950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.862952 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:35.863029 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.863030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.863790 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.863791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.864684 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.864687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.865151 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.865153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.865218 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.865237 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.865422 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.865424 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.865479 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.865522 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":136,"threads_started":1}
I20260926 09:22:35.868536 972 raft_consensus.cc:1261] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.868620 832 raft_consensus.cc:1261] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.868870 1076 consensus_queue.cc:1035] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.869082 1054 consensus_queue.cc:1035] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.878598 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.875716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60635 (local address 127.24.235.65:46775)
0926 09:22:35.875753 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.875755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.875764 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:35.875822 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.875822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.875834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.875890 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.876074 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.876537 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.877701 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.878001 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.878042 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.878298 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.878331 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.878380 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.878462 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.878483 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:22:35.878597 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.875547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46775 (local address 127.24.235.66:60635)
0926 09:22:35.875706 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:35.875718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:35.875766 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:35.875935 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:35.875936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:35.875937 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 09:22:35.876009 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.876011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.876622 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.876623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.877593 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:35.877597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.878096 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.878097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:35.878181 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.878200 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.878401 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.878403 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:35.878453 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:22:35.878501 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 09:22:35.890115 533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.890242 533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.890995 533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Sending DeleteTablet for 3 replicas of tablet 6724d7834ca2461292f14e01246839ec
I20260926 09:22:35.891284 672 tablet_service.cc:1514] Processing DeleteTablet for tablet 6724d7834ca2461292f14e01246839ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:37520
I20260926 09:22:35.891391 1070 tablet_replica.cc:330] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: stopping tablet replica
I20260926 09:22:35.891412 952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6724d7834ca2461292f14e01246839ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 09:22:35.891444 1070 raft_consensus.cc:2229] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:35.891521 1070 raft_consensus.cc:2258] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.891506 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 6724d7834ca2461292f14e01246839ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260926 09:22:35.891603 1069 tablet_replica.cc:330] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: stopping tablet replica
I20260926 09:22:35.891602 1068 tablet_replica.cc:330] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: stopping tablet replica
I20260926 09:22:35.891669 1068 raft_consensus.cc:2229] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.891673 1069 raft_consensus.cc:2229] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.891716 1069 raft_consensus.cc:2258] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.891714 1068 raft_consensus.cc:2258] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.891847 1070 ts_tablet_manager.cc:1905] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.895483 1070 ts_tablet_manager.cc:1918] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:35.895563 1070 log.cc:1196] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/6724d7834ca2461292f14e01246839ec
I20260926 09:22:35.895792 1070 ts_tablet_manager.cc:1939] T 6724d7834ca2461292f14e01246839ec P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting consensus metadata
I20260926 09:22:35.896173 522 catalog_manager.cc:4841] TS 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775): tablet 6724d7834ca2461292f14e01246839ec (table test-table [id=ebe8e13fb73b46738f63d2f1f437985c]) successfully deleted
I20260926 09:22:35.896279 1069 ts_tablet_manager.cc:1905] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.897295 1068 ts_tablet_manager.cc:1905] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.898146 1069 ts_tablet_manager.cc:1918] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:35.898222 1069 log.cc:1196] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/6724d7834ca2461292f14e01246839ec
I20260926 09:22:35.898449 1069 ts_tablet_manager.cc:1939] T 6724d7834ca2461292f14e01246839ec P bda94b25fd82420ea7a720bbb05deece: Deleting consensus metadata
I20260926 09:22:35.899072 522 catalog_manager.cc:4841] TS bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839): tablet 6724d7834ca2461292f14e01246839ec (table test-table [id=ebe8e13fb73b46738f63d2f1f437985c]) successfully deleted
I20260926 09:22:35.899098 1068 ts_tablet_manager.cc:1918] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:35.899156 1068 log.cc:1196] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/6724d7834ca2461292f14e01246839ec
I20260926 09:22:35.899369 1068 ts_tablet_manager.cc:1939] T 6724d7834ca2461292f14e01246839ec P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting consensus metadata
I20260926 09:22:35.899883 521 catalog_manager.cc:4841] TS 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425): tablet 6724d7834ca2461292f14e01246839ec (table test-table [id=ebe8e13fb73b46738f63d2f1f437985c]) successfully deleted
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 09:22:35.926021 584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:35.918771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.24.235.126:46737)
0926 09:22:35.918820 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.918823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.919015 (+ 192us) server_negotiation.cc:407] Connection header received
0926 09:22:35.919080 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.919081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.919095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.919162 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.919899 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.920695 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.921872 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.922172 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.922243 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.925947 (+ 3704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59160: BlockingRecv error: recv got EOF from 127.0.0.1:59160 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 09:22:35 dist-test-slave-7zlw krb5kdc[490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM
I20260926 09:22:35.936537 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.927324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.24.235.126:46737)
0926 09:22:35.927402 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.927404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.927477 (+ 73us) server_negotiation.cc:407] Connection header received
0926 09:22:35.927548 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.927549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.927561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.927623 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:35.928244 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.929048 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.930117 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.930426 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.930542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.933420 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:35.933422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:35.933422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:35.933431 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:35.935200 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.935638 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.935639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.935639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.935701 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:35.935938 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:35.935939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:35.935939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:35.936081 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:35.936186 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.936350 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.936435 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 09:22:35.938664 533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:35.938925 533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:35.940315 812 tablet_service.cc:1467] Processing CreateTablet for tablet b9709d335785493cb76536c545e39ea7 (DEFAULT_TABLE table=test-table [id=0de06102b3d345d4a426cb6636a48591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.940351 672 tablet_service.cc:1467] Processing CreateTablet for tablet b9709d335785493cb76536c545e39ea7 (DEFAULT_TABLE table=test-table [id=0de06102b3d345d4a426cb6636a48591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.940466 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9709d335785493cb76536c545e39ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.940479 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9709d335785493cb76536c545e39ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.941949 952 tablet_service.cc:1467] Processing CreateTablet for tablet b9709d335785493cb76536c545e39ea7 (DEFAULT_TABLE table=test-table [id=0de06102b3d345d4a426cb6636a48591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.941998 1047 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Bootstrap starting.
I20260926 09:22:35.942070 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9709d335785493cb76536c545e39ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.942358 1047 tablet_bootstrap.cc:654] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.943181 1047 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: No bootstrap required, opened a new log
I20260926 09:22:35.943228 1047 ts_tablet_manager.cc:1397] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.943329 1052 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Bootstrap starting.
I20260926 09:22:35.943334 1047 raft_consensus.cc:348] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.943403 1047 raft_consensus.cc:374] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.943423 1047 raft_consensus.cc:729] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.943466 1047 consensus_queue.cc:260] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.943567 1047 ts_tablet_manager.cc:1428] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.943804 1052 tablet_bootstrap.cc:654] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.944121 1046 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Bootstrap starting.
I20260926 09:22:35.944502 1046 tablet_bootstrap.cc:654] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.945269 1046 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: No bootstrap required, opened a new log
I20260926 09:22:35.945325 1046 ts_tablet_manager.cc:1397] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:35.945480 1046 raft_consensus.cc:348] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.945565 1046 raft_consensus.cc:374] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.945586 1046 raft_consensus.cc:729] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda94b25fd82420ea7a720bbb05deece, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.945634 1046 consensus_queue.cc:260] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.945727 1046 ts_tablet_manager.cc:1428] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.946486 1052 tablet_bootstrap.cc:492] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: No bootstrap required, opened a new log
I20260926 09:22:35.946539 1052 ts_tablet_manager.cc:1397] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 09:22:35.946673 1052 raft_consensus.cc:348] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.946738 1052 raft_consensus.cc:374] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.946758 1052 raft_consensus.cc:729] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.946820 1052 consensus_queue.cc:260] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.946906 1052 ts_tablet_manager.cc:1428] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.958149 1076 raft_consensus.cc:482] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:35.958250 1076 raft_consensus.cc:504] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.958397 1076 leader_election.cc:288] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.958598 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9709d335785493cb76536c545e39ea7" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" is_pre_election: true
I20260926 09:22:35.958695 832 raft_consensus.cc:2454] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:35.958855 625 leader_election.cc:302] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:35.958935 1076 raft_consensus.cc:2781] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:35.958961 1076 raft_consensus.cc:482] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:35.958998 1076 raft_consensus.cc:3037] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.959095 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9709d335785493cb76536c545e39ea7" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece" is_pre_election: true
I20260926 09:22:35.959215 972 raft_consensus.cc:2454] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:35.959853 1076 raft_consensus.cc:504] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.959995 1076 leader_election.cc:288] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Requested vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:35.960237 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9709d335785493cb76536c545e39ea7" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece"
I20260926 09:22:35.960243 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9709d335785493cb76536c545e39ea7" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
I20260926 09:22:35.960319 972 raft_consensus.cc:3037] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.960319 832 raft_consensus.cc:3037] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:35.961158 832 raft_consensus.cc:2454] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:35.961282 972 raft_consensus.cc:2454] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:35.961318 625 leader_election.cc:302] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:35.961459 1076 raft_consensus.cc:2781] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:35.961508 1076 raft_consensus.cc:686] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Becoming Leader. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Running, Role: LEADER
I20260926 09:22:35.961560 1076 consensus_queue.cc:237] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.962105 534 catalog_manager.cc:5495] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65). New cstate: current_term: 1 leader_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:35.973184 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:35.969935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.24.235.65:46775)
0926 09:22:35.970012 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:35.970014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:35.970024 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:35.970052 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:35.970053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:35.970064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:35.970132 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:35.970313 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.971111 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.972214 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:35.972496 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:35.972573 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:35.972757 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:35.972793 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:35.972849 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:35.973084 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 09:22:35.973103 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 09:22:35.974009 972 raft_consensus.cc:1261] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.974046 832 raft_consensus.cc:1261] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:35.974252 1076 consensus_queue.cc:1035] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.974346 1054 consensus_queue.cc:1035] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:35.978078 533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.978199 533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:35.978816 533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Sending DeleteTablet for 3 replicas of tablet b9709d335785493cb76536c545e39ea7
I20260926 09:22:35.979077 672 tablet_service.cc:1514] Processing DeleteTablet for tablet b9709d335785493cb76536c545e39ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:37520
I20260926 09:22:35.979113 952 tablet_service.cc:1514] Processing DeleteTablet for tablet b9709d335785493cb76536c545e39ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 09:22:35.979168 1070 tablet_replica.cc:330] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: stopping tablet replica
I20260926 09:22:35.979215 1070 raft_consensus.cc:2229] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:35.979215 1069 tablet_replica.cc:330] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: stopping tablet replica
I20260926 09:22:35.979264 1069 raft_consensus.cc:2229] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.979269 1070 raft_consensus.cc:2258] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.979302 1069 raft_consensus.cc:2258] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.979482 812 tablet_service.cc:1514] Processing DeleteTablet for tablet b9709d335785493cb76536c545e39ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:35 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260926 09:22:35.979573 1068 tablet_replica.cc:330] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: stopping tablet replica
I20260926 09:22:35.979627 1068 raft_consensus.cc:2229] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:35.979671 1068 raft_consensus.cc:2258] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:35.979784 1070 ts_tablet_manager.cc:1905] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.980028 533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:35.980181 533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:35.981345 952 tablet_service.cc:1467] Processing CreateTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 (DEFAULT_TABLE table=test-table [id=56b68e1bee1a43e59a019b44b75b48d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.981419 672 tablet_service.cc:1467] Processing CreateTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 (DEFAULT_TABLE table=test-table [id=56b68e1bee1a43e59a019b44b75b48d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.981501 952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f22b23abd7341bbb5e56dede1e0eca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.981779 1070 ts_tablet_manager.cc:1918] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.981832 1070 log.cc:1196] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/b9709d335785493cb76536c545e39ea7
I20260926 09:22:35.982057 1070 ts_tablet_manager.cc:1939] T b9709d335785493cb76536c545e39ea7 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Deleting consensus metadata
I20260926 09:22:35.982187 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f22b23abd7341bbb5e56dede1e0eca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.982827 1046 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Bootstrap starting.
I20260926 09:22:35.983304 1052 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Bootstrap starting.
I20260926 09:22:35.983484 1046 tablet_bootstrap.cc:654] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.983706 1052 tablet_bootstrap.cc:654] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.984089 812 tablet_service.cc:1467] Processing CreateTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 (DEFAULT_TABLE table=test-table [id=56b68e1bee1a43e59a019b44b75b48d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:35.984236 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f22b23abd7341bbb5e56dede1e0eca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:35.984995 1052 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: No bootstrap required, opened a new log
I20260926 09:22:35.985062 1052 ts_tablet_manager.cc:1397] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:22:35.985139 522 catalog_manager.cc:4841] TS 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775): tablet b9709d335785493cb76536c545e39ea7 (table test-table [id=0de06102b3d345d4a426cb6636a48591]) successfully deleted
I20260926 09:22:35.985222 1052 raft_consensus.cc:348] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.985299 1052 raft_consensus.cc:374] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.985337 1052 raft_consensus.cc:729] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.985378 1052 consensus_queue.cc:260] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.985464 1052 ts_tablet_manager.cc:1428] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.986434 1047 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Bootstrap starting.
I20260926 09:22:35.986807 1047 tablet_bootstrap.cc:654] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:35.987413 1047 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: No bootstrap required, opened a new log
I20260926 09:22:35.987460 1047 ts_tablet_manager.cc:1397] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:35.987560 1047 raft_consensus.cc:348] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.987630 1047 raft_consensus.cc:374] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.987649 1047 raft_consensus.cc:729] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8ae85299f4a98843fe71b2ed3fdc8, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.987686 1047 consensus_queue.cc:260] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.987771 1047 ts_tablet_manager.cc:1428] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:35.988391 1068 ts_tablet_manager.cc:1905] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.989861 1069 ts_tablet_manager.cc:1905] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:35.990008 1068 ts_tablet_manager.cc:1918] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.990061 1068 log.cc:1196] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/b9709d335785493cb76536c545e39ea7
I20260926 09:22:35.990284 1068 ts_tablet_manager.cc:1939] T b9709d335785493cb76536c545e39ea7 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting consensus metadata
I20260926 09:22:35.991523 1069 ts_tablet_manager.cc:1918] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:35.991590 1069 log.cc:1196] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/b9709d335785493cb76536c545e39ea7
I20260926 09:22:35.991818 1069 ts_tablet_manager.cc:1939] T b9709d335785493cb76536c545e39ea7 P bda94b25fd82420ea7a720bbb05deece: Deleting consensus metadata
I20260926 09:22:35.992039 521 catalog_manager.cc:4841] TS 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425): tablet b9709d335785493cb76536c545e39ea7 (table test-table [id=0de06102b3d345d4a426cb6636a48591]) successfully deleted
I20260926 09:22:35.992313 1046 tablet_bootstrap.cc:492] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: No bootstrap required, opened a new log
I20260926 09:22:35.992363 1046 ts_tablet_manager.cc:1397] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260926 09:22:35.992511 1046 raft_consensus.cc:348] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.992581 1046 raft_consensus.cc:374] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:35.992600 1046 raft_consensus.cc:729] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda94b25fd82420ea7a720bbb05deece, State: Initialized, Role: FOLLOWER
I20260926 09:22:35.992640 1046 consensus_queue.cc:260] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:35.992724 1046 ts_tablet_manager.cc:1428] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:35.993242 522 catalog_manager.cc:4841] TS bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839): tablet b9709d335785493cb76536c545e39ea7 (table test-table [id=0de06102b3d345d4a426cb6636a48591]) successfully deleted
I20260926 09:22:36.128341 1076 raft_consensus.cc:482] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:36.128453 1076 raft_consensus.cc:504] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:36.128734 1076 leader_election.cc:288] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:36.128973 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f22b23abd7341bbb5e56dede1e0eca0" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece" is_pre_election: true
I20260926 09:22:36.128950 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f22b23abd7341bbb5e56dede1e0eca0" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" is_pre_election: true
I20260926 09:22:36.129087 832 raft_consensus.cc:2454] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:36.129102 972 raft_consensus.cc:2454] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 0.
I20260926 09:22:36.129271 625 leader_election.cc:302] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, 88c8ae85299f4a98843fe71b2ed3fdc8; no voters:
I20260926 09:22:36.129370 1076 raft_consensus.cc:2781] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:36.129422 1076 raft_consensus.cc:482] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:36.129447 1076 raft_consensus.cc:3037] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:36.130216 1076 raft_consensus.cc:504] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:36.130379 1076 leader_election.cc:288] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Requested vote from peers 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425), bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839)
I20260926 09:22:36.130554 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f22b23abd7341bbb5e56dede1e0eca0" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda94b25fd82420ea7a720bbb05deece"
I20260926 09:22:36.130666 972 raft_consensus.cc:3037] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:36.130668 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f22b23abd7341bbb5e56dede1e0eca0" candidate_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8"
I20260926 09:22:36.130764 832 raft_consensus.cc:3037] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:36.131454 972 raft_consensus.cc:2454] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:36.131599 832 raft_consensus.cc:2454] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecf5ac26a214c5e8e9f0afbd5a4eafe in term 1.
I20260926 09:22:36.131675 626 leader_election.cc:302] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, bda94b25fd82420ea7a720bbb05deece; no voters:
I20260926 09:22:36.131789 1076 raft_consensus.cc:2781] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:36.131837 1076 raft_consensus.cc:686] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Becoming Leader. State: Replica: 2ecf5ac26a214c5e8e9f0afbd5a4eafe, State: Running, Role: LEADER
I20260926 09:22:36.131883 1076 consensus_queue.cc:237] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } }
I20260926 09:22:36.132457 533 catalog_manager.cc:5495] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65). New cstate: current_term: 1 leader_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecf5ac26a214c5e8e9f0afbd5a4eafe" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260926 09:22:36.173971 972 raft_consensus.cc:1261] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:36.173996 832 raft_consensus.cc:1261] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Refusing update from remote peer 2ecf5ac26a214c5e8e9f0afbd5a4eafe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:36.174234 1076 consensus_queue.cc:1035] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda94b25fd82420ea7a720bbb05deece" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:36.174360 1054 consensus_queue.cc:1035] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [LEADER]: Connected to new peer: Peer: permanent_uuid: "88c8ae85299f4a98843fe71b2ed3fdc8" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:36.185287 534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:36.185395 534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:36.186081 534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8b40bd64344f4a9cd7cb463a0f199a: Sending DeleteTablet for 3 replicas of tablet 8f22b23abd7341bbb5e56dede1e0eca0
I20260926 09:22:36.186347 672 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:36 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:37520
I20260926 09:22:36.186375 952 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:36 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42642
I20260926 09:22:36.186481 1069 tablet_replica.cc:330] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: stopping tablet replica
I20260926 09:22:36.186488 1070 tablet_replica.cc:330] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe: stopping tablet replica
I20260926 09:22:36.186501 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f22b23abd7341bbb5e56dede1e0eca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:36 UTC) from {username='oryx', principal='oryx/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:50872
I20260926 09:22:36.186528 1069 raft_consensus.cc:2229] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:36.186566 1069 raft_consensus.cc:2258] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:36.186576 1070 raft_consensus.cc:2229] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:36.186635 1070 raft_consensus.cc:2258] T 8f22b23abd7341bbb5e56dede1e0eca0 P 2ecf5ac26a214c5e8e9f0afbd5a4eafe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:36.186758 1068 tablet_replica.cc:330] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: stopping tablet replica
I20260926 09:22:36.186844 1068 raft_consensus.cc:2229] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:36.186883 1068 raft_consensus.cc:2258] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:36.186880 1069 ts_tablet_manager.cc:1905] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:36.187346 1068 ts_tablet_manager.cc:1905] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:36.188475 1069 ts_tablet_manager.cc:1918] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:36.188494 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 607
I20260926 09:22:36.188544 1069 log.cc:1196] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/8f22b23abd7341bbb5e56dede1e0eca0
I20260926 09:22:36.189659 1069 ts_tablet_manager.cc:1939] T 8f22b23abd7341bbb5e56dede1e0eca0 P bda94b25fd82420ea7a720bbb05deece: Deleting consensus metadata
I20260926 09:22:36.190433 522 catalog_manager.cc:4841] TS bda94b25fd82420ea7a720bbb05deece (127.24.235.67:34839): tablet 8f22b23abd7341bbb5e56dede1e0eca0 (table test-table [id=56b68e1bee1a43e59a019b44b75b48d8]) successfully deleted
I20260926 09:22:36.190510 1068 ts_tablet_manager.cc:1918] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 09:22:36.190555 1068 log.cc:1196] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/8f22b23abd7341bbb5e56dede1e0eca0
I20260926 09:22:36.190709 1068 ts_tablet_manager.cc:1939] T 8f22b23abd7341bbb5e56dede1e0eca0 P 88c8ae85299f4a98843fe71b2ed3fdc8: Deleting consensus metadata
I20260926 09:22:36.190982 521 catalog_manager.cc:4841] TS 88c8ae85299f4a98843fe71b2ed3fdc8 (127.24.235.66:37425): tablet 8f22b23abd7341bbb5e56dede1e0eca0 (table test-table [id=56b68e1bee1a43e59a019b44b75b48d8]) successfully deleted
W20260926 09:22:36.195475 522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:46775 (error 108)
I20260926 09:22:36.195670 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 747
I20260926 09:22:36.197233 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:36.195858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46775 (local address 127.0.0.1:37544)
0926 09:22:36.197135 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:36.197188 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1255,"thread_start_us":38,"threads_started":1}
W20260926 09:22:36.197464 522 catalog_manager.cc:4584] TS 2ecf5ac26a214c5e8e9f0afbd5a4eafe (127.24.235.65:46775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f22b23abd7341bbb5e56dede1e0eca0: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46775: connect: Connection refused (error 111)
I20260926 09:22:36.202203 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 887
I20260926 09:22:36.207274 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 503
2026-09-26T09:22:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2882 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:36 dist-test-slave-7zlw krb5kdc[1133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:36 dist-test-slave-7zlw krb5kdc[1133](info): set up 2 sockets
Sep 26 09:22:36 dist-test-slave-7zlw krb5kdc[1133](info): commencing operation
krb5kdc: starting...
W20260926 09:22:37.266090 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:37 dist-test-slave-7zlw krb5kdc[1133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:37.397786 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:38619
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:37.482366 1146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:37.482707 1146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:37.482734 1146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:37.483982 1146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:37.484037 1146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:37.484054 1146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:37.484069 1146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:37.484083 1146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:37.485674 1146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:37.485939 1146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:37.486241 1146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:37.491580 1154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:37.491720 1151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:37.491819 1146 server_base.cc:1034] running on GCE node
W20260926 09:22:37.491592 1152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:37.492125 1146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:37.492429 1146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:37.493603 1146 hybrid_clock.cc:648] HybridClock initialized: now 1790414557493584 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T09:22:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1287 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:37 dist-test-slave-7zlw krb5kdc[1159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:37 dist-test-slave-7zlw krb5kdc[1159](info): set up 2 sockets
Sep 26 09:22:37 dist-test-slave-7zlw krb5kdc[1159](info): commencing operation
krb5kdc: starting...
W20260926 09:22:38.565763 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:38 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:38.693406 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45073
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:38.774982 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:38.775206 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:38.775228 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:38.776304 1172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:38.776341 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:38.776355 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:38.776367 1172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:38.776379 1172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:38.777813 1172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:38.778018 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:38.778198 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:38.783571 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:38.783600 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:38.783581 1172 server_base.cc:1034] running on GCE node
W20260926 09:22:38.783599 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:38.784147 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:38.784477 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:38.785621 1172 hybrid_clock.cc:648] HybridClock initialized: now 1790414558785604 us; error 43 us; skew 500 ppm
Sep 26 09:22:38 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414558, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:38.788210 1172 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:38.788642 1172 webserver.cc:460] Webserver started at http://127.24.235.126:42673/ using document root <none> and password file <none>
I20260926 09:22:38.788916 1172 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:38.788986 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:38.789206 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:38.790295 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "a89daff83cb0424b89e2dcae30b41179"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "0ebfd6527a8aa5b749d6b6af885d1529"
server_key_iv: "0bced81367863ad4b81bab5a570b9d9f"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.790665 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "a89daff83cb0424b89e2dcae30b41179"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "0ebfd6527a8aa5b749d6b6af885d1529"
server_key_iv: "0bced81367863ad4b81bab5a570b9d9f"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.792006 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:38.792835 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:38.793074 1172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:38.793175 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "a89daff83cb0424b89e2dcae30b41179"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "0ebfd6527a8aa5b749d6b6af885d1529"
server_key_iv: "0bced81367863ad4b81bab5a570b9d9f"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.793280 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:38.803583 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:38.805749 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:38.805925 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:38.810037 1172 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:45073
I20260926 09:22:38.810067 1239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:45073 every 8 connection(s)
I20260926 09:22:38.810418 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:38.811272 1240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:38.813664 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Bootstrap starting.
I20260926 09:22:38.814258 1240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:38.814555 1240 log.cc:824] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:38.815234 1240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: No bootstrap required, opened a new log
I20260926 09:22:38.816911 1240 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:38.817116 1240 raft_consensus.cc:374] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:38.817142 1240 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89daff83cb0424b89e2dcae30b41179, State: Initialized, Role: FOLLOWER
I20260926 09:22:38.817312 1240 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:38.817374 1240 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:38.817413 1240 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:38.817461 1240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:38.817960 1240 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:38.818040 1240 leader_election.cc:302] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89daff83cb0424b89e2dcae30b41179; no voters:
I20260926 09:22:38.818218 1240 leader_election.cc:288] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:38.818269 1243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:38.818478 1240 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:38.818666 1243 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 LEADER]: Becoming Leader. State: Replica: a89daff83cb0424b89e2dcae30b41179, State: Running, Role: LEADER
I20260926 09:22:38.818763 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1172
I20260926 09:22:38.818771 1243 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:38.818868 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:38.819092 25517 external_mini_cluster.cc:1428] Setting key 2495fc7850a08f9d63fc9c85a2773f03
I20260926 09:22:38.819144 1244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a89daff83cb0424b89e2dcae30b41179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } } }
I20260926 09:22:38.819221 1244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:38.819329 1245 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89daff83cb0424b89e2dcae30b41179. Latest consensus state: current_term: 1 leader_uuid: "a89daff83cb0424b89e2dcae30b41179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } } }
I20260926 09:22:38.819388 1245 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:38.819450 1252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:38.819979 1252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:38.821730 1252 catalog_manager.cc:1347] Generated new cluster ID: 7fbdb91f13db4ffa82f856abdacd2bc5
I20260926 09:22:38.821781 1252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:22:38 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:38.831360 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:38.819678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.24.235.126:45073)
0926 09:22:38.819960 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:38.819971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:38.820002 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:22:38.820119 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:38.820126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:38.820310 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:38.820456 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:38.821214 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:38.822655 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:38.823753 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:38.824368 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:38.824462 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:38.827403 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:38.827410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:38.827411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:38.827429 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:38.829583 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:38.830005 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:38.830009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:38.830009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:38.830147 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:38.830353 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:38.830354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:38.830355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:38.830716 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:38.830876 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:38.830955 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 09:22:38.831117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260926 09:22:38.838255 1252 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:38.838793 1252 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:38.844743 1252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Generated new TSK 0
I20260926 09:22:38.844971 1252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:38.877589 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:38.961560 1266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:38.961781 1266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:38.961805 1266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:38.963056 1266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:38.963104 1266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:38.963130 1266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:38.964602 1266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:38.964872 1266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:38.965153 1266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:38.970367 1274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:38.970367 1271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:38.970386 1272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:38.970728 1266 server_base.cc:1034] running on GCE node
I20260926 09:22:38.970934 1266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:38.971210 1266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:38.972357 1266 hybrid_clock.cc:648] HybridClock initialized: now 1790414558972338 us; error 36 us; skew 500 ppm
Sep 26 09:22:38 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414558, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:38.974884 1266 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:38.975340 1266 webserver.cc:460] Webserver started at http://127.24.235.65:34733/ using document root <none> and password file <none>
I20260926 09:22:38.975540 1266 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:38.975581 1266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:38.975682 1266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:38.976652 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "cf312964216e4814ab2b7460e1f19beb"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "20d00ce83d78b454f22bfcea42a6b695"
server_key_iv: "c3dba82803e6e48772eba93d457d3fef"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.976948 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "cf312964216e4814ab2b7460e1f19beb"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "20d00ce83d78b454f22bfcea42a6b695"
server_key_iv: "c3dba82803e6e48772eba93d457d3fef"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.978266 1266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:38.978973 1281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:38.979156 1266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:38.979257 1266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "cf312964216e4814ab2b7460e1f19beb"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "20d00ce83d78b454f22bfcea42a6b695"
server_key_iv: "c3dba82803e6e48772eba93d457d3fef"
server_key_version: "encryptionkey@0"
I20260926 09:22:38.979331 1266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:38.987787 1266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:38.990077 1266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:38.990242 1266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:38.990468 1266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:38.990782 1266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:38.990839 1266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:38.990881 1266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:38.990909 1266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:38.997092 1266 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44885
I20260926 09:22:38.997138 1394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44885 every 8 connection(s)
I20260926 09:22:38.997539 1266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414558, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:39.007501 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1266
I20260926 09:22:39.007627 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:39.007843 25517 external_mini_cluster.cc:1428] Setting key 0afa26c217529e7ed801d6c0688c9cbf
I20260926 09:22:39.010421 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:38.998391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:57247 (local address 127.24.235.126:45073)
0926 09:22:38.998533 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:38.998535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:38.998854 (+ 319us) server_negotiation.cc:407] Connection header received
0926 09:22:38.999544 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:38.999546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:38.999558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:38.999587 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.000932 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.001713 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.003105 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.003411 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.003452 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.006433 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:39.006434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:39.006435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:39.006445 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:39.008768 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.009365 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.009366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.009367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.009446 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.009759 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.009760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.009760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.009871 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:39.009925 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:39.010275 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 09:22:39.010354 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 09:22:39.010738 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:38.998441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:57247)
0926 09:22:38.998706 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:38.998749 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:38.999422 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:38.999817 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:38.999821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.000535 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.000822 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.000828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.001829 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.001830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.003006 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.003008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.003558 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.003570 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.003725 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.004347 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.004358 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.006319 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:39.008854 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.008861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.008864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.009213 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.009526 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.009527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.009527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.009685 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.009940 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:39.009945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:39.010126 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 09:22:39.010253 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 09:22:39.010443 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260926 09:22:39.011354 1395 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45073
I20260926 09:22:39.011482 1395 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:39.011699 1395 heartbeater.cc:511] Master 127.24.235.126:45073 requested a full tablet report, sending...
I20260926 09:22:39.012281 1204 ts_manager.cc:194] Registered new tserver with Master: cf312964216e4814ab2b7460e1f19beb (127.24.235.65:44885)
I20260926 09:22:39.012915 1204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:57247
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:39.037127 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:39.118995 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.119235 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.119274 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.120483 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.120528 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.120555 1400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:39.121943 1400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.122124 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.122303 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:39.127887 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.128072 1400 server_base.cc:1034] running on GCE node
W20260926 09:22:39.127897 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.127912 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.128378 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.128633 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.129777 1400 hybrid_clock.cc:648] HybridClock initialized: now 1790414559129749 us; error 41 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.132208 1400 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:39.132668 1400 webserver.cc:460] Webserver started at http://127.24.235.66:39691/ using document root <none> and password file <none>
I20260926 09:22:39.132915 1400 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.132961 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.133117 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:39.134217 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "4396577df0a845da94b31e9ac0b0dabb"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "d96c800e171c4a0fa06ff65b0080398b"
server_key_iv: "7b93b4103b91073e7bba25dda3f66cfc"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.134563 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "4396577df0a845da94b31e9ac0b0dabb"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "d96c800e171c4a0fa06ff65b0080398b"
server_key_iv: "7b93b4103b91073e7bba25dda3f66cfc"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.136059 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:39.136888 1415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.137117 1400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:39.137210 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "4396577df0a845da94b31e9ac0b0dabb"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "d96c800e171c4a0fa06ff65b0080398b"
server_key_iv: "7b93b4103b91073e7bba25dda3f66cfc"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.137296 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.151023 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.153527 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.153746 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.154042 1400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:39.154426 1400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:39.154488 1400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.154529 1400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:39.154583 1400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.161440 1400 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:38015
I20260926 09:22:39.161485 1528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:38015 every 8 connection(s)
I20260926 09:22:39.161854 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:39.162364 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1400
I20260926 09:22:39.162496 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:39.162748 25517 external_mini_cluster.cc:1428] Setting key f346aa243d3660258a45dc712aaa13a1
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
I20260926 09:22:39.172935 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.162805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:43829 (local address 127.24.235.126:45073)
0926 09:22:39.162865 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.162867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.163169 (+ 302us) server_negotiation.cc:407] Connection header received
0926 09:22:39.163793 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.163794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.163806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.163870 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.164856 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.165492 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.166528 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.166830 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.166906 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.169502 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:39.169504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:39.169504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:39.169513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:39.171418 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.171881 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.171883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.171883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.171947 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.172197 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.172197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.172198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.172356 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:39.172440 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:39.172791 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 09:22:39.172868 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 09:22:39.173141 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.162774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:43829)
0926 09:22:39.162983 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.163028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.163686 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.163923 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.163926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.164502 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.164736 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.164742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.165605 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.165606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.166422 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.166424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.166921 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.166922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.167033 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.167617 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.167628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.169401 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:39.171546 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.171553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.171555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.171790 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.172020 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.172021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.172021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.172126 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.172455 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:39.172462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:39.172655 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 09:22:39.172753 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:22:39.172877 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260926 09:22:39.173659 1529 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45073
I20260926 09:22:39.173743 1529 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:39.173889 1529 heartbeater.cc:511] Master 127.24.235.126:45073 requested a full tablet report, sending...
I20260926 09:22:39.174249 1204 ts_manager.cc:194] Registered new tserver with Master: 4396577df0a845da94b31e9ac0b0dabb (127.24.235.66:38015)
I20260926 09:22:39.174665 1204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:43829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:39.193743 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:39.276722 1534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.276930 1534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.276953 1534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.278141 1534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.278185 1534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.278211 1534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:39.279619 1534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.279793 1534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.279963 1534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:39.285113 1542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.285130 1540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.285301 1539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.285254 1534 server_base.cc:1034] running on GCE node
I20260926 09:22:39.285535 1534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.285795 1534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.286968 1534 hybrid_clock.cc:648] HybridClock initialized: now 1790414559286961 us; error 33 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.289494 1534 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:39.289950 1534 webserver.cc:460] Webserver started at http://127.24.235.67:43575/ using document root <none> and password file <none>
I20260926 09:22:39.290185 1534 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.290232 1534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.290382 1534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:39.291419 1534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "0dedf67659464355af62e4aef1dece10"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "e3846a60caf30a3185e58a176ce8e3ad"
server_key_iv: "3fd271d87b7c5ea09b3b06f7aae6569b"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.291775 1534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "0dedf67659464355af62e4aef1dece10"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "e3846a60caf30a3185e58a176ce8e3ad"
server_key_iv: "3fd271d87b7c5ea09b3b06f7aae6569b"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.293244 1534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:22:39.293951 1549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.294158 1534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:39.294261 1534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "0dedf67659464355af62e4aef1dece10"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "e3846a60caf30a3185e58a176ce8e3ad"
server_key_iv: "3fd271d87b7c5ea09b3b06f7aae6569b"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.294334 1534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.311352 1534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.313691 1534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.313872 1534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.314131 1534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:39.314466 1534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:39.314522 1534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.314566 1534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:39.314590 1534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.321520 1534 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:35587
I20260926 09:22:39.321574 1662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:35587 every 8 connection(s)
I20260926 09:22:39.321941 1534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:39.329357 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1534
I20260926 09:22:39.329486 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:39.329679 25517 external_mini_cluster.cc:1428] Setting key c9ae404ae0d9201bafcfa03d46c2c987
I20260926 09:22:39.332190 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.322752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:54351 (local address 127.24.235.126:45073)
0926 09:22:39.322814 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.322816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.323020 (+ 204us) server_negotiation.cc:407] Connection header received
0926 09:22:39.323527 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.323528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.323537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.323633 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.324638 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.325243 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.326328 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.326516 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.326576 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.328957 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:39.328959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:39.328959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:39.328967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:39.330722 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.331133 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.331134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.331135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.331195 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.331495 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.331496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.331497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.331624 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:39.331715 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:39.332011 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 09:22:39.332098 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:22:39.332403 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.322706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:54351)
0926 09:22:39.322884 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.322927 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.323418 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.323700 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.323704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.324305 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.324517 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.324522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.325358 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.325359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.326230 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.326233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.326619 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.326620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.326755 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.327400 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.327408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.328854 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:39.330796 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.330804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.330806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.331041 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.331297 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:39.331298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:39.331298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:39.331420 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:39.331707 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:39.331714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:39.331868 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 09:22:39.331978 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 09:22:39.332149 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260926 09:22:39.332880 1663 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45073
I20260926 09:22:39.333045 1663 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:39.333237 1663 heartbeater.cc:511] Master 127.24.235.126:45073 requested a full tablet report, sending...
I20260926 09:22:39.333631 1204 ts_manager.cc:194] Registered new tserver with Master: 0dedf67659464355af62e4aef1dece10 (127.24.235.67:35587)
I20260926 09:22:39.333968 1204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:54351
I20260926 09:22:39.341881 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
Entry for principal oryx/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab.
WARNING: no policy specified for oryx/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
Entry for principal oryx/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab.
WARNING: no policy specified for oryx/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
Entry for principal oryx/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab.
I20260926 09:22:39.425077 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1266
I20260926 09:22:39.430397 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1400
I20260926 09:22:39.435416 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1534
I20260926 09:22:39.440425 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1172
I20260926 09:22:39.446122 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45073
--webserver_interface=127.24.235.126
--webserver_port=42673
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:39.528734 1672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.528971 1672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.528996 1672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.530339 1672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:39.530392 1672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.530407 1672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.530421 1672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:39.530432 1672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:39.532006 1672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.532289 1672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.532572 1672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:39.538000 1672 server_base.cc:1034] running on GCE node
W20260926 09:22:39.537963 1678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.537961 1677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.537977 1680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.538455 1672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.538767 1672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.539949 1672 hybrid_clock.cc:648] HybridClock initialized: now 1790414559539946 us; error 38 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.542706 1672 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:39.543273 1672 webserver.cc:460] Webserver started at http://127.24.235.126:42673/ using document root <none> and password file <none>
I20260926 09:22:39.543514 1672 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.543599 1672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.545140 1672 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:39.545845 1687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.546013 1672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:39.546116 1672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "a89daff83cb0424b89e2dcae30b41179"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "0ebfd6527a8aa5b749d6b6af885d1529"
server_key_iv: "0bced81367863ad4b81bab5a570b9d9f"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.546494 1672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.553732 1672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.555774 1672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.555975 1672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.560500 1672 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:45073
I20260926 09:22:39.560578 1739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:45073 every 8 connection(s)
I20260926 09:22:39.560968 1672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:39.561311 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1672
I20260926 09:22:39.561628 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:44885
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:45073
--webserver_port=34733
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:22:39.562626 1740 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:22:39.563290 1740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Bootstrap starting.
I20260926 09:22:39.574397 1740 log.cc:824] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:39.576721 1740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:22:39.577109 1740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Bootstrap complete.
I20260926 09:22:39.580379 1740 raft_consensus.cc:348] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:39.580708 1740 raft_consensus.cc:729] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89daff83cb0424b89e2dcae30b41179, State: Initialized, Role: FOLLOWER
I20260926 09:22:39.580866 1740 consensus_queue.cc:260] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:39.580931 1740 raft_consensus.cc:388] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:39.580968 1740 raft_consensus.cc:482] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:39.581030 1740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:22:39.582511 1740 raft_consensus.cc:504] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:39.582635 1740 leader_election.cc:302] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a89daff83cb0424b89e2dcae30b41179; no voters:
I20260926 09:22:39.582800 1740 leader_election.cc:288] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:22:39.582893 1744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:22:39.583055 1740 sys_catalog.cc:564] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:39.583061 1744 raft_consensus.cc:686] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [term 2 LEADER]: Becoming Leader. State: Replica: a89daff83cb0424b89e2dcae30b41179, State: Running, Role: LEADER
I20260926 09:22:39.583282 1744 consensus_queue.cc:237] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } }
I20260926 09:22:39.583585 1745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a89daff83cb0424b89e2dcae30b41179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } } }
I20260926 09:22:39.583593 1744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a89daff83cb0424b89e2dcae30b41179. Latest consensus state: current_term: 2 leader_uuid: "a89daff83cb0424b89e2dcae30b41179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a89daff83cb0424b89e2dcae30b41179" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45073 } } }
I20260926 09:22:39.583766 1744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:39.583756 1745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:39.584071 1753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:39.584854 1753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:39.584990 1753 catalog_manager.cc:1259] Loaded cluster ID: 7fbdb91f13db4ffa82f856abdacd2bc5
I20260926 09:22:39.585042 1753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:39.586395 1753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:39.586592 1753 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a89daff83cb0424b89e2dcae30b41179: Loaded TSK: 0
I20260926 09:22:39.586807 1753 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:22:39.652307 1742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.652519 1742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.652545 1742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.653801 1742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.653846 1742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.653872 1742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:39.655319 1742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.65.keytab
--principal=oryx/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:44885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=34733
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.655570 1742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.655808 1742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:39.661834 1769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.661834 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.661916 1742 server_base.cc:1034] running on GCE node
W20260926 09:22:39.661864 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.662333 1742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.662642 1742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.663820 1742 hybrid_clock.cc:648] HybridClock initialized: now 1790414559663805 us; error 32 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.666733 1742 init.cc:377] Logged in from keytab as oryx/127.24.235.65@KRBTEST.COM (short username oryx)
I20260926 09:22:39.667296 1742 webserver.cc:460] Webserver started at http://127.24.235.65:34733/ using document root <none> and password file <none>
I20260926 09:22:39.667577 1742 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.667662 1742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.669473 1742 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:22:39.670181 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.670426 1742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:39.670513 1742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "cf312964216e4814ab2b7460e1f19beb"
format_stamp: "Formatted at 2026-09-26 09:22:38 on dist-test-slave-7zlw"
server_key: "20d00ce83d78b454f22bfcea42a6b695"
server_key_iv: "c3dba82803e6e48772eba93d457d3fef"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.670928 1742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.689522 1742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.692118 1742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.692366 1742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.692648 1742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:39.692972 1742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:39.693005 1742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.693084 1742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:39.693111 1742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.699734 1742 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44885
I20260926 09:22:39.699783 1889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44885 every 8 connection(s)
I20260926 09:22:39.700150 1742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:39.707875 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1742
I20260926 09:22:39.708254 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:38015
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:45073
--webserver_port=39691
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.714288 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.700916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:45161)
0926 09:22:39.701129 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.701189 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.701666 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.705750 (+ 4084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.705754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.706360 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.706560 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.706564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.708230 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.708231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.709637 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.709640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.710325 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.710327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.710459 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.711019 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.711028 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.714061 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
W20260926 09:22:39.714783 1890 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:45073 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.235.126:45073: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
W20260926 09:22:39.714855 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.700992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:45161 (local address 127.24.235.126:45073)
0926 09:22:39.705215 (+ 4223us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.705227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.705276 (+ 49us) server_negotiation.cc:407] Connection header received
0926 09:22:39.705417 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.705424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.705495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.705681 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.706700 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.708107 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.709746 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.710207 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.710348 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.714642 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:45161: BlockingRecv error: recv got EOF from 127.24.235.65:45161 (error 108)
Metrics: {"server-negotiator.queue_time_us":4160,"thread_start_us":105,"threads_started":1}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.722318 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.715020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:58227)
0926 09:22:39.715155 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.715183 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.715302 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.715721 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.715722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.715901 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.715963 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.715965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.717107 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.717108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.718150 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.718153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.718923 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.718924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.719013 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.719588 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.719593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.722207 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 09:22:39.722652 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.715514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:58227 (local address 127.24.235.126:45073)
0926 09:22:39.715556 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.715559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.715567 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:39.715594 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.715595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.715610 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.715674 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.716053 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.716993 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.718433 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.718835 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.718917 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.722320 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:58227: BlockingRecv error: recv got EOF from 127.24.235.65:58227 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.729323 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.722579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:41035)
0926 09:22:39.722766 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.722778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.722833 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.723132 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.723133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.723349 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.723584 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.723592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.724534 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.724535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.725606 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.725610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.726346 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.726355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.726449 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.726958 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.726963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.729226 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
W20260926 09:22:39.729457 1890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:22:39.729818 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.722768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:41035 (local address 127.24.235.126:45073)
0926 09:22:39.723014 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.723017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.723026 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:39.723049 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.723050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.723060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.723150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.723716 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.724435 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.725698 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.726108 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.726183 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.729726 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:41035: BlockingRecv error: recv got EOF from 127.24.235.65:41035 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":84,"threads_started":1}
W20260926 09:22:39.810214 1894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.810475 1894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.810513 1894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.811710 1894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.811785 1894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.811831 1894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:39.813397 1894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.66.keytab
--principal=oryx/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:38015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=39691
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.813678 1894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.813917 1894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:39.819384 1904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.819469 1902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.819384 1901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.819675 1894 server_base.cc:1034] running on GCE node
I20260926 09:22:39.819904 1894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.820158 1894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.821332 1894 hybrid_clock.cc:648] HybridClock initialized: now 1790414559821329 us; error 39 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.824040 1894 init.cc:377] Logged in from keytab as oryx/127.24.235.66@KRBTEST.COM (short username oryx)
I20260926 09:22:39.824669 1894 webserver.cc:460] Webserver started at http://127.24.235.66:39691/ using document root <none> and password file <none>
I20260926 09:22:39.824921 1894 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.825001 1894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.826462 1894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:39.827255 1911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.827500 1894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:39.827616 1894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "4396577df0a845da94b31e9ac0b0dabb"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "d96c800e171c4a0fa06ff65b0080398b"
server_key_iv: "7b93b4103b91073e7bba25dda3f66cfc"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.828060 1894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.840981 1894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.844012 1894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.844239 1894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.844530 1894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:39.844911 1894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:39.844980 1894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.845078 1894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:39.845125 1894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.851855 1894 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:38015
I20260926 09:22:39.851897 2024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:38015 every 8 connection(s)
I20260926 09:22:39.852309 1894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:39.855298 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 1894
I20260926 09:22:39.855688 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:35587
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:45073
--webserver_port=43575
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.861562 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.853196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:59063)
0926 09:22:39.853444 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.853484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.854004 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.854255 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.854259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.854858 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.855069 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.855074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.856150 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.856152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.857296 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.857300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.858054 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.858055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.858200 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.858906 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.858916 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.861394 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
W20260926 09:22:39.861634 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.853218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:59063 (local address 127.24.235.126:45073)
0926 09:22:39.853300 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.853303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.853660 (+ 357us) server_negotiation.cc:407] Connection header received
0926 09:22:39.854108 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.854109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.854133 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.854194 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.855197 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.855992 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.857531 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.857870 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.857965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.861532 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:59063: BlockingRecv error: recv got EOF from 127.24.235.66:59063 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 09:22:39.861841 2025 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:45073 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.235.126:45073: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.867750 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.862128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:44037)
0926 09:22:39.862291 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.862305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.862377 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.862530 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.862531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.862738 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.862805 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.862807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.863528 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.863530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.864503 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.864507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.864924 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.864926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.865004 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.865511 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.865517 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.867657 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
W20260926 09:22:39.867822 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.862188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:44037 (local address 127.24.235.126:45073)
0926 09:22:39.862258 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.862260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.862374 (+ 114us) server_negotiation.cc:407] Connection header received
0926 09:22:39.862445 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.862445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.862455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.862511 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.862907 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.863435 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.864615 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.864813 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.864886 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.867749 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:44037: BlockingRecv error: recv got EOF from 127.24.235.66:44037 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.873229 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.868116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:58547)
0926 09:22:39.868234 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.868245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.868300 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.868434 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.868435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.868596 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.868637 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.868639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.869390 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.869390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.870246 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.870248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.870629 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.870630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.870693 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.871144 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.871149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.873131 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260926 09:22:39.873277 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.868181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:58547 (local address 127.24.235.126:45073)
0926 09:22:39.868234 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.868236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.868284 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:22:39.868357 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.868358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.868366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.868413 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.868735 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.869309 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.870356 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.870550 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.870615 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.873211 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:58547: BlockingRecv error: recv got EOF from 127.24.235.66:58547 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 09:22:39.873423 2025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:22:39.942471 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:39.942724 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:39.942765 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:39.943974 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:39.944046 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:39.944096 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:39.945690 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/krb5kdc/oryx_127.24.235.67.keytab
--principal=oryx/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:35587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=43575
--tserver_master_addrs=127.24.235.126:45073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:39.945969 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:39.946215 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:39.951903 2034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:39.951954 2037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.951954 2028 server_base.cc:1034] running on GCE node
W20260926 09:22:39.952080 2035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:39.952324 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:39.952618 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:39.953789 2028 hybrid_clock.cc:648] HybridClock initialized: now 1790414559953777 us; error 30 us; skew 500 ppm
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414559, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:39.956913 2028 init.cc:377] Logged in from keytab as oryx/127.24.235.67@KRBTEST.COM (short username oryx)
I20260926 09:22:39.957540 2028 webserver.cc:460] Webserver started at http://127.24.235.67:43575/ using document root <none> and password file <none>
I20260926 09:22:39.957811 2028 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:39.957870 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:39.959404 2028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:22:39.960090 2044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.960292 2028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:22:39.960405 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "0dedf67659464355af62e4aef1dece10"
format_stamp: "Formatted at 2026-09-26 09:22:39 on dist-test-slave-7zlw"
server_key: "e3846a60caf30a3185e58a176ce8e3ad"
server_key_iv: "3fd271d87b7c5ea09b3b06f7aae6569b"
server_key_version: "encryptionkey@0"
I20260926 09:22:39.960829 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:39.979521 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:39.982044 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:39.982246 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:39.982581 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:39.982940 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:39.983007 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.983062 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:39.983093 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:39.989717 2028 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:35587
I20260926 09:22:39.989766 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:35587 every 8 connection(s)
I20260926 09:22:39.990152 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:39.991896 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2028
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:39.998920 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.990902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:41419)
0926 09:22:39.991092 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.991135 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.991614 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.991919 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.991923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.992716 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.993051 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.993057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.993869 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.993870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.995015 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.995017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.995653 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.995654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:39.995806 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.996365 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:39.996376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:39.998769 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
W20260926 09:22:39.998963 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.990920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:41419 (local address 127.24.235.126:45073)
0926 09:22:39.991048 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.991050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.991236 (+ 186us) server_negotiation.cc:407] Connection header received
0926 09:22:39.991736 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.991738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.991754 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.991846 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.993136 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.993773 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.995162 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.995564 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.995631 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.998888 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:41419: BlockingRecv error: recv got EOF from 127.24.235.67:41419 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 09:22:39.999217 2158 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:45073 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.235.126:45073: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
I20260926 09:22:40.000494 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.992555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.24.235.126:45073)
0926 09:22:39.992609 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.992611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.992737 (+ 126us) server_negotiation.cc:407] Connection header received
0926 09:22:39.992799 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.992800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.992809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.992870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.993618 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.994199 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.995239 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:39.995527 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:39.995631 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:39.997144 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:39.997148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:39.997148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:39.997159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:39.998979 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.999394 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.999398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.999398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.999521 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:39.999740 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:39.999741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:39.999741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:39.999977 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:40.000127 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:40.000272 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 09:22:40.000389 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:39 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:40.003693 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:39.999436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:60387)
0926 09:22:39.999475 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:39.999486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:39.999525 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:39.999638 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:39.999639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:39.999757 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:39.999803 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:39.999804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.000416 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.000417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.001124 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.001126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.001598 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.001599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.001648 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.001982 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:40.001986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:40.003640 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 09:22:40.003928 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:39.999465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:60387 (local address 127.24.235.126:45073)
0926 09:22:39.999534 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:39.999536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:39.999547 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:39.999571 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:39.999572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:39.999581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:39.999626 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:39.999887 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.000341 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.001231 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.001533 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.001597 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.003872 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:60387: BlockingRecv error: recv got EOF from 127.24.235.67:60387 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:40.008136 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:40.003962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:55035)
0926 09:22:40.004008 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:40.004018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:40.004060 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:40.004207 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:40.004207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:40.004309 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:40.004339 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.004340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.005153 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.005153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.005632 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.005633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.005997 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.005998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.006038 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.006459 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:40.006464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:40.008082 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:22:40.008315 2158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:22:40.008334 1893 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:40.004050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:55035 (local address 127.24.235.126:45073)
0926 09:22:40.004090 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:40.004092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:40.004100 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:40.004123 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:40.004124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:40.004132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:40.004201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:40.004429 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.004972 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.005725 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.005920 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.005985 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.008269 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:55035: BlockingRecv error: recv got EOF from 127.24.235.67:55035 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:40.735373 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:40.729984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:57265)
0926 09:22:40.730127 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:40.730146 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:40.730202 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:40.730391 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:40.730392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:40.730576 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:40.730624 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.730627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.731406 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.731408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.731954 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.731957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.732520 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.732521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.732591 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.733166 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:40.733171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:40.735313 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
W20260926 09:22:40.735538 2163 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:40.730090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:57265 (local address 127.24.235.126:45073)
0926 09:22:40.730247 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:40.730249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:40.730259 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:40.730285 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:40.730285 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:40.730296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:40.730368 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:40.730731 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.731314 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.732080 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.732430 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.732498 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.735461 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:57265: BlockingRecv error: recv got EOF from 127.24.235.65:57265 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:40 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:40.879436 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:40.873955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:50963)
0926 09:22:40.874125 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:40.874142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:40.874204 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:40.874390 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:40.874392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:40.874691 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:40.874746 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.874748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.875572 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.875574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.876415 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:40.876418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.876741 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.876742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:40.876785 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.877387 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:40.877392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:40.879360 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
W20260926 09:22:40.879530 2163 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:40.874110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:50963 (local address 127.24.235.126:45073)
0926 09:22:40.874157 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:40.874159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:40.874190 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:22:40.874253 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:40.874253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:40.874264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:40.874329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:40.874867 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.875445 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.876485 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:40.876673 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:40.876728 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:40.879481 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:50963: BlockingRecv error: recv got EOF from 127.24.235.66:50963 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:41.014144 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:41.008796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:55841)
0926 09:22:41.008976 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:41.008998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:41.009128 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:41.009324 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:41.009325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:41.009555 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:41.009616 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.009619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.010319 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.010320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.010976 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.010978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.011350 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.011351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.011399 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.012021 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:41.012027 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:41.014034 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260926 09:22:41.014191 2163 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:41.008914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:55841 (local address 127.24.235.126:45073)
0926 09:22:41.008974 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:41.008976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:41.009111 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:22:41.009221 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:41.009222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:41.009235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:41.009314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:41.009708 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.010205 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.011070 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.011263 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.011323 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.014121 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:55841: BlockingRecv error: recv got EOF from 127.24.235.67:55841 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:41.748246 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:41.735906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:52585)
0926 09:22:41.736043 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:41.736064 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:41.736119 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:41.736351 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:41.736352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:41.736510 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:41.736604 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.736607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.737309 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.737310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.737861 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.737862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.738200 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.738201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.738245 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.746196 (+ 7951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:41.746202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:41.748172 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260926 09:22:41.748394 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:41.736049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:52585 (local address 127.24.235.126:45073)
0926 09:22:41.736196 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:41.736198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:41.736207 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:41.736231 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:41.736232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:41.736243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:41.736313 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:41.736695 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.737236 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.737937 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.738125 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.738200 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.748299 (+ 10099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:52585: BlockingRecv error: recv got EOF from 127.24.235.65:52585 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:41 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:41.885366 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:41.880093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:55357)
0926 09:22:41.880261 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:41.880272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:41.880324 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:41.880520 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:41.880521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:41.880718 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:41.880769 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.880771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.881563 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.881564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.882097 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:41.882100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.882493 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.882494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:41.882561 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.883108 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:41.883113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:41.885264 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260926 09:22:41.885416 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:41.880225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:55357 (local address 127.24.235.126:45073)
0926 09:22:41.880295 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:41.880301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:41.880312 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:41.880391 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:41.880392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:41.880403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:41.880466 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:41.880873 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.881459 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.882205 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:41.882396 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:41.882460 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:41.885350 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:55357: BlockingRecv error: recv got EOF from 127.24.235.66:55357 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:42.019729 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:42.014815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:59805)
0926 09:22:42.015002 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:42.015018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:42.015097 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:42.015283 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:42.015284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:42.015461 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:42.015509 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.015511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.016143 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.016144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.017062 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.017065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.017419 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.017419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.017457 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.017939 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:42.017943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:42.019669 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
W20260926 09:22:42.019829 2167 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:42.014989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:59805 (local address 127.24.235.126:45073)
0926 09:22:42.015078 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:42.015080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:42.015093 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:42.015163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:42.015164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:42.015177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:42.015221 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:42.015579 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.016072 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.017152 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.017354 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.017411 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.019761 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:59805: BlockingRecv error: recv got EOF from 127.24.235.67:59805 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:42.753978 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:42.748876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:48553)
0926 09:22:42.749081 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:42.749101 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:42.749165 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:42.749332 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:42.749333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:42.749509 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:42.749559 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.749560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.750426 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.750427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.751087 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.751090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.751579 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.751580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.751624 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.752146 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:42.752159 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:42.753917 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
W20260926 09:22:42.754093 2171 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:42.748903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:48553 (local address 127.24.235.126:45073)
0926 09:22:42.749073 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:42.749076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:42.749166 (+ 90us) server_negotiation.cc:407] Connection header received
0926 09:22:42.749234 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:42.749235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:42.749257 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:42.749312 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:42.749659 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.750345 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.751163 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.751504 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.751563 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.754015 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:48553: BlockingRecv error: recv got EOF from 127.24.235.65:48553 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:42 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:42.890712 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:42.885887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:53569)
0926 09:22:42.886096 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:42.886109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:42.886220 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:42.886399 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:42.886401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:42.886585 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:42.886632 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.886634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.887334 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.887336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.887860 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:42.887861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.888259 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.888260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:42.888318 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.888801 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:42.888805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:42.890651 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260926 09:22:42.890792 2171 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:42.885985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:53569 (local address 127.24.235.126:45073)
0926 09:22:42.886081 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:42.886084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:42.886194 (+ 110us) server_negotiation.cc:407] Connection header received
0926 09:22:42.886311 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:42.886312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:42.886326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:42.886390 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:42.886761 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.887248 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.887945 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:42.888138 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:42.888259 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:42.890746 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:53569: BlockingRecv error: recv got EOF from 127.24.235.66:53569 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:43.024994 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:43.020255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:38469)
0926 09:22:43.020393 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:43.020404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:43.020463 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:43.020594 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:43.020595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:43.020755 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:43.020804 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.020805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.021497 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.021498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.022016 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.022017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.022412 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.022413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.022471 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.022983 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:43.022988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:43.024926 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260926 09:22:43.025246 2171 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:43.020332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:38469 (local address 127.24.235.126:45073)
0926 09:22:43.020418 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:43.020420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:43.020448 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:22:43.020516 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:43.020517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:43.020528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:43.020577 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:43.020863 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.021431 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.022119 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.022317 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.022389 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.025185 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:38469: BlockingRecv error: recv got EOF from 127.24.235.67:38469 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:43.759341 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:43.754566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:55399)
0926 09:22:43.754742 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:43.754762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:43.754830 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:43.754965 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:43.754966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:43.755129 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:43.755183 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.755185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.755850 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.755850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.756372 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.756373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.756837 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.756837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.756904 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.757503 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:43.757511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:43.759279 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
W20260926 09:22:43.759500 2175 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:43.754622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55399 (local address 127.24.235.126:45073)
0926 09:22:43.754773 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:43.754775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:43.754825 (+ 50us) server_negotiation.cc:407] Connection header received
0926 09:22:43.754875 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:43.754875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:43.754885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:43.754943 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:43.755249 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.755776 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.756446 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.756747 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.756833 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.759395 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:55399: BlockingRecv error: recv got EOF from 127.24.235.65:55399 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:43 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:43.896353 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:43.891246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:41413)
0926 09:22:43.891486 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:43.891500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:43.891577 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:43.891710 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:43.891711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:43.891906 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:43.891956 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.891958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.892637 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.892638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.893220 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:43.893222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.893595 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.893596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:43.893666 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.894207 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:43.894212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:43.896291 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":123,"threads_started":1}
W20260926 09:22:43.896555 2175 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:43.891347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:41413 (local address 127.24.235.126:45073)
0926 09:22:43.891472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:43.891475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:43.891557 (+ 82us) server_negotiation.cc:407] Connection header received
0926 09:22:43.891636 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:43.891637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:43.891650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:43.891703 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:43.892045 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.892548 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.893314 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:43.893508 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:43.893586 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:43.896476 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:41413: BlockingRecv error: recv got EOF from 127.24.235.66:41413 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:44.030328 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:44.025551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:37273)
0926 09:22:44.025872 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:44.025884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:44.025949 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:44.026062 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:44.026063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:44.026242 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:44.026290 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.026292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.026926 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.026926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.027464 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.027466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.027800 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.027801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.027843 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.028388 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:44.028400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:44.030228 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":258,"threads_started":1}
W20260926 09:22:44.030368 2175 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:44.025740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37273 (local address 127.24.235.126:45073)
0926 09:22:44.025801 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:44.025803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:44.025948 (+ 145us) server_negotiation.cc:407] Connection header received
0926 09:22:44.025992 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:44.025992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:44.026004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:44.026056 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:44.026346 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.026854 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.027535 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.027732 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.027784 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.030304 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:37273: BlockingRecv error: recv got EOF from 127.24.235.67:37273 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:44.765643 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:44.760049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:53941)
0926 09:22:44.760225 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:44.760238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:44.760304 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:44.760550 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:44.760552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:44.760746 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:44.760800 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.760802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.761550 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.761551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.762112 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.762114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.762588 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.762589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.762632 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.763298 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:44.763303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:44.765581 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":121,"threads_started":1}
W20260926 09:22:44.765820 2179 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:44.760127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:53941 (local address 127.24.235.126:45073)
0926 09:22:44.760290 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:44.760292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:44.760310 (+ 18us) server_negotiation.cc:407] Connection header received
0926 09:22:44.760359 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:44.760360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:44.760370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:44.760441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:44.760918 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.761460 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.762233 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.762459 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.762581 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.765737 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:53941: BlockingRecv error: recv got EOF from 127.24.235.65:53941 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:44 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:44.902110 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:44.897066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:40511)
0926 09:22:44.897249 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:44.897265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:44.897331 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:44.897528 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:44.897530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:44.897773 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:44.897846 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.897849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.898648 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.898649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.899194 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:44.899195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.899644 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.899645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:44.899694 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.900162 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:44.900167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:44.902049 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
W20260926 09:22:44.902278 2179 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:44.897228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:40511 (local address 127.24.235.126:45073)
0926 09:22:44.897308 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:44.897310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:44.897322 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:44.897426 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:44.897427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:44.897440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:44.897496 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:44.897944 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.898522 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.899297 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:44.899502 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:44.899576 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:44.902185 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:40511: BlockingRecv error: recv got EOF from 127.24.235.66:40511 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:45.036010 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:45.030933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:37325)
0926 09:22:45.031074 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:45.031087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:45.031162 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:45.031375 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:45.031376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:45.031641 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:45.031691 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.031693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.032479 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.032480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.033003 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.033005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.033481 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.033482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.033533 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.034095 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:45.034100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:45.035938 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
W20260926 09:22:45.036226 2179 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:45.031000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37325 (local address 127.24.235.126:45073)
0926 09:22:45.031063 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:45.031066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:45.031164 (+ 98us) server_negotiation.cc:407] Connection header received
0926 09:22:45.031262 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:45.031263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:45.031278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:45.031340 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:45.031815 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.032404 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.033169 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.033374 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.033499 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.036133 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:37325: BlockingRecv error: recv got EOF from 127.24.235.67:37325 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:45.771711 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:45.766265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:33395)
0926 09:22:45.766435 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:45.766518 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:45.766610 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:45.766908 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:45.766910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:45.767126 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:45.767176 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.767178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.767957 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.767958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.768632 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.768635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.769070 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.769071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.769123 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.769616 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:45.769620 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:45.771617 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
W20260926 09:22:45.771773 2183 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:45.766344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:33395 (local address 127.24.235.126:45073)
0926 09:22:45.766501 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:45.766503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:45.766693 (+ 190us) server_negotiation.cc:407] Connection header received
0926 09:22:45.766748 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:45.766749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:45.766764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:45.766843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:45.767263 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.767858 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.768767 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.768959 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.769055 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.771719 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:33395: BlockingRecv error: recv got EOF from 127.24.235.65:33395 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":106,"threads_started":1}
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:45 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:45.908087 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:45.902767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:60101)
0926 09:22:45.902934 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:45.902954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:45.903020 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:45.903155 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:45.903156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:45.903511 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:45.903564 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.903567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.904349 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.904350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.904936 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:45.904938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.905502 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.905503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:45.905593 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.906189 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:45.906207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:45.908015 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
W20260926 09:22:45.908318 2183 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:45.902883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60101 (local address 127.24.235.126:45073)
0926 09:22:45.902954 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:45.902956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:45.903003 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:22:45.903079 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:45.903079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:45.903091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:45.903132 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:45.903684 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.904242 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.905059 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:45.905392 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:45.905480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:45.908239 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:60101: BlockingRecv error: recv got EOF from 127.24.235.66:60101 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:46.041829 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:46.036591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:56335)
0926 09:22:46.036744 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:46.036754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:46.036802 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:46.036959 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:46.036960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:46.037152 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:46.037201 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.037202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.037958 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.037960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.038499 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.038501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.038885 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.038886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.038928 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.039410 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:46.039415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:46.041765 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
W20260926 09:22:46.041970 2183 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:46.036748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:56335 (local address 127.24.235.126:45073)
0926 09:22:46.036823 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:46.036825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:46.036854 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:22:46.036876 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:46.036877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:46.036888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:46.036933 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:46.037303 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.037856 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.038604 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.038810 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.038872 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.041879 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:56335: BlockingRecv error: recv got EOF from 127.24.235.67:56335 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:46.783571 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:46.772302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:60889)
0926 09:22:46.772473 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:46.772495 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:46.772563 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:46.772779 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:46.772780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:46.772953 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:46.773003 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.773005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.773751 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.773752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.774305 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.774307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.774754 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.774755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.774840 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.781322 (+ 6482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:46.781328 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:46.783458 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
W20260926 09:22:46.783666 2187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:46.772459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:60889 (local address 127.24.235.126:45073)
0926 09:22:46.772640 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:46.772642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:46.772651 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:46.772675 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:46.772675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:46.772686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:46.772755 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:46.773137 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.773632 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.774401 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.774626 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.774714 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.783582 (+ 8868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:60889: BlockingRecv error: recv got EOF from 127.24.235.65:60889 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":126,"threads_started":1}
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:46 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:46.914202 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:46.908885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:59541)
0926 09:22:46.909120 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:46.909140 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:46.909203 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:46.909422 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:46.909424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:46.909638 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:46.909693 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.909695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.910374 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.910375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.910952 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:46.910954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.911294 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.911295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:46.911361 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.911952 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:46.911957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:46.914123 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20260926 09:22:46.914271 2187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:46.909079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:59541 (local address 127.24.235.126:45073)
0926 09:22:46.909261 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:46.909264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:46.909275 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:46.909310 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:46.909311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:46.909322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:46.909376 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:46.909810 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.910300 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.911039 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:46.911231 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:46.911286 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:46.914225 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:59541: BlockingRecv error: recv got EOF from 127.24.235.66:59541 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:47.048089 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:47.042449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:48075)
0926 09:22:47.042710 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:47.042724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:47.042807 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:47.043006 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:47.043007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:47.043231 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:47.043309 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.043311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.044044 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.044045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.044755 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.044758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.045143 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.045144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.045203 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.045738 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:47.045742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:47.047989 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":210,"threads_started":1}
W20260926 09:22:47.048152 2187 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:47.042599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:48075 (local address 127.24.235.126:45073)
0926 09:22:47.042708 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:47.042710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:47.042785 (+ 75us) server_negotiation.cc:407] Connection header received
0926 09:22:47.042876 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:47.042877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:47.042911 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:47.042977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:47.043399 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.043943 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.044818 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.045009 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.045124 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.048090 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:48075: BlockingRecv error: recv got EOF from 127.24.235.67:48075 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:47.789004 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:47.784122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:36289)
0926 09:22:47.784277 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:47.784288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:47.784390 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:47.784524 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:47.784524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:47.784687 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:47.784738 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.784740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.785404 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.785404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.785964 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.785966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.786329 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.786330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.786381 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.786872 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:47.786877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:47.788927 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260926 09:22:47.789134 2191 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:47.784254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:36289 (local address 127.24.235.126:45073)
0926 09:22:47.784384 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:47.784386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:47.784396 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:47.784432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:47.784432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:47.784443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:47.784514 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:47.784813 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.785336 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.786041 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.786235 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.786292 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.789074 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:36289: BlockingRecv error: recv got EOF from 127.24.235.65:36289 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:47 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:47.919885 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:47.914853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:39185)
0926 09:22:47.915011 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:47.915027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:47.915103 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:47.915237 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:47.915238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:47.915403 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:47.915454 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.915455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.916338 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.916339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.916965 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:47.916967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.917418 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.917419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:47.917466 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.917998 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:47.918003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:47.919825 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
W20260926 09:22:47.920148 2191 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:47.914970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:39185 (local address 127.24.235.126:45073)
0926 09:22:47.915044 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:47.915046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:47.915086 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:22:47.915149 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:47.915150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:47.915161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:47.915212 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:47.915526 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.916218 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.917117 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:47.917335 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:47.917399 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:47.919948 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:39185: BlockingRecv error: recv got EOF from 127.24.235.66:39185 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:48.053951 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:48.048796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:51403)
0926 09:22:48.048951 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:48.048963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:48.049067 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:48.049277 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:48.049278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:48.049469 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:48.049518 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.049520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.050223 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.050224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.050777 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.050779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.051137 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.051138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.051183 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.051803 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:48.051808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:48.053846 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
W20260926 09:22:48.054052 2191 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:48.048914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:51403 (local address 127.24.235.126:45073)
0926 09:22:48.049004 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:48.049006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:48.049067 (+ 61us) server_negotiation.cc:407] Connection header received
0926 09:22:48.049121 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:48.049122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:48.049138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:48.049214 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:48.049600 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.050113 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.050859 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.051061 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.051118 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.053970 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:51403: BlockingRecv error: recv got EOF from 127.24.235.67:51403 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:48.794492 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:48.789613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:57545)
0926 09:22:48.789766 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:48.789778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:48.789829 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:48.790062 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:48.790063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:48.790231 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:48.790278 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.790279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.790947 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.790948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.791479 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.791481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.791867 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.791868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.791913 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.792371 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:48.792375 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:48.794433 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260926 09:22:48.794617 2195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:48.789792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:57545 (local address 127.24.235.126:45073)
0926 09:22:48.789927 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:48.789929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:48.789937 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:22:48.789963 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:48.789963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:48.789973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:48.790044 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:48.790353 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.790876 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.791565 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.791787 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.791852 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.794533 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:57545: BlockingRecv error: recv got EOF from 127.24.235.65:57545 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:48 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:48.925725 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:48.920610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:44119)
0926 09:22:48.920786 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:48.920798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:48.920874 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:48.921065 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:48.921066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:48.921236 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:48.921285 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.921287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.922076 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.922077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.922618 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:48.922620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.923055 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.923056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:48.923107 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.923620 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:48.923626 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:48.925662 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
W20260926 09:22:48.925933 2195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:48.920782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:44119 (local address 127.24.235.126:45073)
0926 09:22:48.920839 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:48.920841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:48.920854 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:48.920942 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:48.920943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:48.920957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:48.921048 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:48.921398 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.921960 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.922747 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:48.922944 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:48.923012 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:48.925849 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:44119: BlockingRecv error: recv got EOF from 127.24.235.66:44119 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:49.059433 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:49.054673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:34231)
0926 09:22:49.054851 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:49.054862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:49.054921 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:49.055064 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:49.055065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:49.055254 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:49.055303 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.055305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.055981 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.055982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.056533 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.056535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.056912 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.056913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.056963 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.057469 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:49.057473 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:49.059372 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
W20260926 09:22:49.059538 2195 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:49.054818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:34231 (local address 127.24.235.126:45073)
0926 09:22:49.054871 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:49.054873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:49.054910 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:22:49.054967 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:49.054968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:49.054980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:49.055042 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:49.055378 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.055899 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.056616 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.056816 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.056872 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.059468 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:34231: BlockingRecv error: recv got EOF from 127.24.235.67:34231 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:49.801059 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:49.795014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:54937)
0926 09:22:49.795155 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:49.795169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:49.795257 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:49.795461 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:49.795462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:49.795679 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:49.795729 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.795731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.796527 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.796528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.797289 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.797293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.797869 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.797870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.797977 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.798589 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:49.798594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:49.800916 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
W20260926 09:22:49.801172 2199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:49.795111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:54937 (local address 127.24.235.126:45073)
0926 09:22:49.795248 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:49.795250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:49.795263 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:22:49.795360 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:49.795362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:49.795375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:49.795442 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:49.795831 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.796417 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.797384 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.797751 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.797840 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.801058 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:54937: BlockingRecv error: recv got EOF from 127.24.235.65:54937 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:49 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:49.931728 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:49.926363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:34011)
0926 09:22:49.926578 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:49.926591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:49.926653 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:49.926846 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:49.926847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:49.927115 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:49.927178 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.927180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.927855 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.927856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.928451 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:49.928453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.928879 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.928880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:49.928931 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.929571 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:49.929576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:49.931624 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
W20260926 09:22:49.931897 2199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:49.926585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:34011 (local address 127.24.235.126:45073)
0926 09:22:49.926689 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:49.926692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:49.926704 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:49.926735 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:49.926736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:49.926750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:49.926816 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:49.927249 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.927753 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.928543 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:49.928773 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:49.928831 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:49.931801 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:34011: BlockingRecv error: recv got EOF from 127.24.235.66:34011 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:50.065740 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:50.060108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:45055)
0926 09:22:50.060352 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:50.060366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:50.060429 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:50.060616 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:50.060617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:50.060854 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:50.060904 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.060906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.061786 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.061787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.062402 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.062404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.062766 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.062767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.062827 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.063448 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:50.063453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:50.065666 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
W20260926 09:22:50.065886 2199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:50.060272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:45055 (local address 127.24.235.126:45073)
0926 09:22:50.060355 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:50.060357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:50.060429 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:22:50.060508 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:50.060509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:50.060525 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:50.060595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:50.061112 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.061677 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.062484 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.062671 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.062749 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.065802 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:45055: BlockingRecv error: recv got EOF from 127.24.235.67:45055 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:50.807199 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:50.801879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:34995)
0926 09:22:50.802063 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:50.802076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:50.802147 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:50.802353 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:50.802355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:50.802525 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:50.802573 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.802575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.803347 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.803348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.803943 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.803946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.804531 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.804532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.804684 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.805330 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:50.805335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:50.807139 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
W20260926 09:22:50.807420 2203 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:50.802041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:34995 (local address 127.24.235.126:45073)
0926 09:22:50.802191 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:50.802194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:50.802204 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:50.802239 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:50.802239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:50.802251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:50.802315 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:50.802712 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.803228 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.804089 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.804432 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.804516 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.807339 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:34995: BlockingRecv error: recv got EOF from 127.24.235.65:34995 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:50 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:50.937397 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:50.932341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:37105)
0926 09:22:50.932540 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:50.932552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:50.932653 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:50.932833 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:50.932833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:50.933070 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:50.933120 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.933123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.933872 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.933874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.934422 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:50.934424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.934807 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.934808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:50.934870 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.935361 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:50.935365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:50.937294 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
W20260926 09:22:50.937448 2203 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:50.932451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:37105 (local address 127.24.235.126:45073)
0926 09:22:50.932538 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:50.932540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:50.932633 (+ 93us) server_negotiation.cc:407] Connection header received
0926 09:22:50.932731 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:50.932732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:50.932745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:50.932826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:50.933214 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.933776 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.934518 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:50.934720 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:50.934797 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:50.937383 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:37105: BlockingRecv error: recv got EOF from 127.24.235.66:37105 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:51.072224 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:51.066286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:52569)
0926 09:22:51.066483 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:51.066496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:51.066596 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:51.066799 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:51.066800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:51.066987 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:51.067087 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.067089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.067889 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.067890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.068456 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.068459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.068891 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.068892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.068941 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.069618 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:51.069623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:51.072123 (+ 2500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":133,"threads_started":1}
W20260926 09:22:51.072425 2203 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:51.066399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:52569 (local address 127.24.235.126:45073)
0926 09:22:51.066538 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:51.066540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:51.066551 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:51.066696 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:51.066696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:51.066710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:51.066776 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:51.067198 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.067774 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.068599 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.068802 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.068870 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.072314 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:52569: BlockingRecv error: recv got EOF from 127.24.235.67:52569 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:51.819885 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:51.807782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:60803)
0926 09:22:51.807935 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:51.807956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:51.808011 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:51.808302 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:51.808303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:51.808569 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:51.808621 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.808624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.809423 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.809424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.810000 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.810002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.810526 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.810528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.810605 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.817807 (+ 7202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:51.817814 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:51.819809 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
W20260926 09:22:51.820070 2207 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:51.807936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:60803 (local address 127.24.235.126:45073)
0926 09:22:51.808089 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:51.808091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:51.808108 (+ 17us) server_negotiation.cc:407] Connection header received
0926 09:22:51.808133 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:51.808134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:51.808144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:51.808256 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:51.808744 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.809314 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.810129 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.810423 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.810492 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.819971 (+ 9479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:60803: BlockingRecv error: recv got EOF from 127.24.235.65:60803 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:51 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:51.942794 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:51.937944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:33565)
0926 09:22:51.938098 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:51.938111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:51.938175 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:51.938348 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:51.938349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:51.938516 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:51.938567 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.938568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.939270 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.939271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.939824 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:51.939826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.940158 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.940159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:51.940203 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.940724 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:51.940729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:51.942735 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260926 09:22:51.942991 2207 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:51.938066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:33565 (local address 127.24.235.126:45073)
0926 09:22:51.938128 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:51.938130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:51.938163 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:22:51.938227 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:51.938228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:51.938239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:51.938324 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:51.938657 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.939189 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.939904 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:51.940092 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:51.940146 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:51.942911 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:33565: BlockingRecv error: recv got EOF from 127.24.235.66:33565 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:52.077709 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:52.072793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:36185)
0926 09:22:52.072965 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:52.072984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:52.073106 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:52.073261 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:52.073263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:52.073432 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:52.073480 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.073482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.074207 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.074208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.074781 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.074783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.075121 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.075123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.075170 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.075630 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:52.075635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:52.077645 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260926 09:22:52.077860 2207 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:52.072932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:36185 (local address 127.24.235.126:45073)
0926 09:22:52.073009 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:52.073052 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:52.073087 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:22:52.073150 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:52.073151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:52.073165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:52.073228 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:52.073553 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.074120 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.074859 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.075035 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.075091 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.077780 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:36185: BlockingRecv error: recv got EOF from 127.24.235.67:36185 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:52.825927 2210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:52.820467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:55361)
0926 09:22:52.820627 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:52.820639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:52.820705 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:52.820884 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:52.820885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:52.821166 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:52.821211 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.821213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.821987 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.821988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.822517 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.822519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.822895 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.822897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.822964 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.823452 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:52.823459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:52.825809 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
W20260926 09:22:52.825997 2211 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:52.820520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55361 (local address 127.24.235.126:45073)
0926 09:22:52.820636 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:52.820639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:52.820686 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:22:52.820775 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:52.820776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:52.820789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:52.820877 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:52.821296 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.821859 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.822606 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.822805 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.822895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.825915 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:55361: BlockingRecv error: recv got EOF from 127.24.235.65:55361 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:52 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:52.948328 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:52.943325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:60517)
0926 09:22:52.943516 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:52.943528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:52.943594 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:52.943733 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:52.943734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:52.943894 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:52.943944 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.943946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.944623 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.944624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.945196 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:52.945197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.945733 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.945734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:52.945778 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.946423 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:52.946429 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:52.948265 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
W20260926 09:22:52.948621 2211 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:52.943419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60517 (local address 127.24.235.126:45073)
0926 09:22:52.943473 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:52.943475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:52.943575 (+ 100us) server_negotiation.cc:407] Connection header received
0926 09:22:52.943655 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:52.943656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:52.943669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:52.943724 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:52.944039 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.944546 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.945339 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:52.945620 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:52.945696 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:52.948552 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:60517: BlockingRecv error: recv got EOF from 127.24.235.66:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:53.083233 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:53.078202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:45029)
0926 09:22:53.078401 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:53.078418 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:53.078491 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:53.078648 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:53.078649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:53.078889 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:53.078949 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.078951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.079654 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.079656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.080191 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.080193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.080550 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.080551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.080622 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.081231 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:53.081235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:53.083143 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
W20260926 09:22:53.083297 2211 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:53.078276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:45029 (local address 127.24.235.126:45073)
0926 09:22:53.078348 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:53.078350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:53.078476 (+ 126us) server_negotiation.cc:407] Connection header received
0926 09:22:53.078547 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:53.078548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:53.078565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:53.078637 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:53.079034 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.079530 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.080292 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.080477 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.080552 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.083226 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:45029: BlockingRecv error: recv got EOF from 127.24.235.67:45029 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:53.831234 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:53.826546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:47531)
0926 09:22:53.826705 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:53.826717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:53.826782 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:53.826971 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:53.826972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:53.827143 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:53.827198 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.827200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.827876 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.827877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.828427 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.828428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.828763 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.828764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.828804 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.829383 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:53.829387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:53.831176 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
W20260926 09:22:53.831421 2215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:53.826688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:47531 (local address 127.24.235.126:45073)
0926 09:22:53.826816 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:53.826818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:53.826828 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:53.826860 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:53.826861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:53.826871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:53.826949 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:53.827294 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.827800 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.828501 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.828697 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.828748 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.831264 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:47531: BlockingRecv error: recv got EOF from 127.24.235.65:47531 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:53 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:53.954362 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:53.949123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:41617)
0926 09:22:53.949298 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:53.949309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:53.949391 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:53.949561 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:53.949562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:53.949753 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:53.949802 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.949803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.950574 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.950574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.951088 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:53.951089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.951464 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.951464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:53.951502 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.952058 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:53.952063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:53.954295 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260926 09:22:53.954434 2215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:53.949192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:41617 (local address 127.24.235.126:45073)
0926 09:22:53.949276 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:53.949278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:53.949398 (+ 120us) server_negotiation.cc:407] Connection header received
0926 09:22:53.949458 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:53.949459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:53.949472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:53.949517 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:53.949902 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.950508 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.951205 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:53.951394 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:53.951458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:53.954387 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:41617: BlockingRecv error: recv got EOF from 127.24.235.66:41617 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.67@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:54.088595 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:54.083781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.67:36393)
0926 09:22:54.083937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:54.083946 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:54.083998 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:54.084135 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:54.084136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:54.084314 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:54.084361 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.084363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.085088 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.085089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.085633 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.085635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.085992 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.085992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.086046 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.086614 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:54.086620 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:54.088535 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":69,"threads_started":1}
W20260926 09:22:54.088754 2215 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:54.083838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:36393 (local address 127.24.235.126:45073)
0926 09:22:54.083910 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:54.083912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:54.083988 (+ 76us) server_negotiation.cc:407] Connection header received
0926 09:22:54.084044 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:54.084045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:54.084058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:54.084118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:54.084427 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.084983 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.085727 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.085910 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.085967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.088669 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.67:36393: BlockingRecv error: recv got EOF from 127.24.235.67:36393 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.65@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:54.836520 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:54.831808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.65:48955)
0926 09:22:54.831948 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:54.831958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:54.832007 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:54.832218 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:54.832218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:54.832391 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:54.832440 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.832442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.833149 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.833151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.833702 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.833704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.834069 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.834070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.834116 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.834670 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:54.834675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:54.836461 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260926 09:22:54.836799 2219 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:54.831918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:48955 (local address 127.24.235.126:45073)
0926 09:22:54.832045 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:54.832047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:54.832057 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:54.832082 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:54.832083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:54.832101 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:54.832167 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:54.832512 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.833049 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.833789 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.833985 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.834051 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.836731 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.65:48955: BlockingRecv error: recv got EOF from 127.24.235.65:48955 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":87,"threads_started":1}
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 09:22:54 dist-test-slave-7zlw krb5kdc[1159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.235.66@KRBTEST.COM for oryx/127.24.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 09:22:54.959542 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:54.954942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45073 (local address 127.24.235.66:49335)
0926 09:22:54.955067 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:54.955077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:54.955124 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:54.955272 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:54.955272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:54.955429 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:54.955482 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.955484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.956132 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.956132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.956645 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:54.956646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.957126 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.957127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:54.957171 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.957718 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:54.957723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:54.959485 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.235.126:45073: Server oryx/127.24.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260926 09:22:54.959679 2219 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:54.955024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:49335 (local address 127.24.235.126:45073)
0926 09:22:54.955119 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:54.955121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:54.955132 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:22:54.955177 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:54.955177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:54.955188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:54.955265 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:54.955547 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.956059 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.956747 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:54.956952 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:54.957114 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:54.959597 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.235.66:49335: BlockingRecv error: recv got EOF from 127.24.235.66:49335 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 09:22:54.996829 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1742
I20260926 09:22:55.003954 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1894
I20260926 09:22:55.008718 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2028
I20260926 09:22:55.013134 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 1672
2026-09-26T09:22:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17517 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T09:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:55Z Disabled control of system clock
I20260926 09:22:55.042135 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:40539
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:40539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 09:22:55.125658 2224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.125875 2224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.125896 2224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.127179 2224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:55.127233 2224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.127246 2224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.127259 2224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:55.127271 2224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:55.128616 2224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:40539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.128831 2224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.129102 2224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:22:55.134742 2224 server_base.cc:1034] running on GCE node
W20260926 09:22:55.134701 2232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.134791 2229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.134755 2230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.135259 2224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.135632 2224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.136819 2224 hybrid_clock.cc:648] HybridClock initialized: now 1790414575136785 us; error 52 us; skew 500 ppm
I20260926 09:22:55.138224 2224 webserver.cc:460] Webserver started at http://127.24.235.126:42539/ using document root <none> and password file <none>
I20260926 09:22:55.138486 2224 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.138581 2224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.138793 2224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.139899 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "31d88b4c1e7146d2b2aae4f36b0ad874"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "d9140e0631ef2488660d5f5551dac4dc"
server_key_iv: "2871f9d9c904c5a4c4ebc685e3e9c6d6"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.140265 2224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "31d88b4c1e7146d2b2aae4f36b0ad874"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "d9140e0631ef2488660d5f5551dac4dc"
server_key_iv: "2871f9d9c904c5a4c4ebc685e3e9c6d6"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.141789 2224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:22:55.142741 2238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.142942 2224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:55.143050 2224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "31d88b4c1e7146d2b2aae4f36b0ad874"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "d9140e0631ef2488660d5f5551dac4dc"
server_key_iv: "2871f9d9c904c5a4c4ebc685e3e9c6d6"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.143146 2224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.153818 2224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.154160 2224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.154325 2224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.158413 2224 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:40539
I20260926 09:22:55.158455 2290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:40539 every 8 connection(s)
I20260926 09:22:55.158838 2224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:55.159590 2291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:55.162426 2291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874: Bootstrap starting.
I20260926 09:22:55.163060 2291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:55.163369 2291 log.cc:824] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:55.164191 2291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874: No bootstrap required, opened a new log
I20260926 09:22:55.166236 2291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } }
I20260926 09:22:55.166393 2291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:55.166419 2291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d88b4c1e7146d2b2aae4f36b0ad874, State: Initialized, Role: FOLLOWER
I20260926 09:22:55.166615 2291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } }
I20260926 09:22:55.166718 2291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:55.166759 2291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:55.166822 2291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:55.166832 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2224
I20260926 09:22:55.166932 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:55.167161 25517 external_mini_cluster.cc:1428] Setting key f33e242c1bc50ea24c27757f7bf0eef6
I20260926 09:22:55.167660 2291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } }
I20260926 09:22:55.167801 2291 leader_election.cc:302] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d88b4c1e7146d2b2aae4f36b0ad874; no voters:
I20260926 09:22:55.168001 2291 leader_election.cc:288] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:55.168035 2296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:55.168165 2296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [term 1 LEADER]: Becoming Leader. State: Replica: 31d88b4c1e7146d2b2aae4f36b0ad874, State: Running, Role: LEADER
I20260926 09:22:55.168329 2291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:55.168349 2296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } }
I20260926 09:22:55.168903 2297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d88b4c1e7146d2b2aae4f36b0ad874. Latest consensus state: current_term: 1 leader_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } } }
I20260926 09:22:55.169106 2297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:55.169437 2301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:55.170029 2301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:55.170452 2298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d88b4c1e7146d2b2aae4f36b0ad874" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40539 } } }
I20260926 09:22:55.170619 2298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:55.171861 2301 catalog_manager.cc:1347] Generated new cluster ID: 8d9b2970e94c4317bd71816b97005dbb
I20260926 09:22:55.171959 2301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:55.173502 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.167730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.24.235.126:40539)
0926 09:22:55.167921 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.167930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.167963 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:22:55.168120 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.168128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.168227 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.168373 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.168597 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.170082 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.171208 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.171778 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.171958 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.172716 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.172721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.172721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.172928 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.172969 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.172984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.173090 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.173193 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.173279 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260926 09:22:55.211313 2301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:55.212085 2301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:55.217355 2301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d88b4c1e7146d2b2aae4f36b0ad874: Generated new TSK 0
I20260926 09:22:55.217646 2301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:55.228971 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--builtin_ntp_servers=127.24.235.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:22:55.331323 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.331583 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.331622 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.332834 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.332907 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.332948 2315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:55.334460 2315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.334750 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.335070 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:55.340688 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.340711 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.340710 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.340794 2315 server_base.cc:1034] running on GCE node
I20260926 09:22:55.341257 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.341563 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.342763 2315 hybrid_clock.cc:648] HybridClock initialized: now 1790414575342737 us; error 36 us; skew 500 ppm
I20260926 09:22:55.344300 2315 webserver.cc:460] Webserver started at http://127.24.235.65:40355/ using document root <none> and password file <none>
I20260926 09:22:55.344659 2315 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.344720 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.344998 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.346367 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "7d92ea2f0c5346ebbd387f9c65d0c482"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "085afdec1c62449c92e51f8e0c388f3e"
server_key_iv: "24ec46c8d4cd6aed4de56ce01693db81"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.346688 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "7d92ea2f0c5346ebbd387f9c65d0c482"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "085afdec1c62449c92e51f8e0c388f3e"
server_key_iv: "24ec46c8d4cd6aed4de56ce01693db81"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.348098 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:55.348909 2329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.349226 2315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:55.349306 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "7d92ea2f0c5346ebbd387f9c65d0c482"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "085afdec1c62449c92e51f8e0c388f3e"
server_key_iv: "24ec46c8d4cd6aed4de56ce01693db81"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.349364 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.366768 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.367087 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.367198 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.367410 2315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:55.367858 2315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:55.367897 2315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.367920 2315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:55.367933 2315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.374353 2315 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:37135
I20260926 09:22:55.374429 2442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:37135 every 8 connection(s)
I20260926 09:22:55.374912 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:55.375495 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2315
I20260926 09:22:55.375603 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:55.375844 25517 external_mini_cluster.cc:1428] Setting key 2270d7c636486eb6b8cf35a42612a514
I20260926 09:22:55.377110 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--builtin_ntp_servers=127.24.235.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 09:22:55.381989 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.376101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:55583 (local address 127.24.235.126:40539)
0926 09:22:55.376156 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.376159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.376363 (+ 204us) server_negotiation.cc:407] Connection header received
0926 09:22:55.376993 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.376995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.377038 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.377117 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.377906 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.378740 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.380248 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.380474 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.380575 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.381649 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.381650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.381651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.381657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.381674 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.381679 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.381741 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.381852 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.381899 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:22:55.382231 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.375995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:40539 (local address 127.24.235.65:55583)
0926 09:22:55.376213 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:55.376266 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:55.376877 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:55.377179 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:55.377183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:55.377208 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:55.377795 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.377800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.378857 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.378859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.380130 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.380133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.380587 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.380589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.380746 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.381477 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:55.381478 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:55.381508 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:55.381535 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:55.381747 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:55.381752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:55.381756 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:55.381855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 09:22:55.381977 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260926 09:22:55.382951 2443 heartbeater.cc:347] Connected to a master server at 127.24.235.126:40539
I20260926 09:22:55.383073 2443 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:55.383315 2443 heartbeater.cc:511] Master 127.24.235.126:40539 requested a full tablet report, sending...
I20260926 09:22:55.383961 2255 ts_manager.cc:194] Registered new tserver with Master: 7d92ea2f0c5346ebbd387f9c65d0c482 (127.24.235.65:37135)
I20260926 09:22:55.384984 2255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.65:55583
W20260926 09:22:55.465021 2446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.465323 2446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.465346 2446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.466497 2446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.466534 2446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.466559 2446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:55.467885 2446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.468140 2446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.468511 2446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:55.474107 2452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.474210 2451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.474107 2454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.474344 2446 server_base.cc:1034] running on GCE node
I20260926 09:22:55.474627 2446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.475014 2446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.476229 2446 hybrid_clock.cc:648] HybridClock initialized: now 1790414575476201 us; error 36 us; skew 500 ppm
I20260926 09:22:55.477846 2446 webserver.cc:460] Webserver started at http://127.24.235.66:42947/ using document root <none> and password file <none>
I20260926 09:22:55.478132 2446 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.478209 2446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.478376 2446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.479528 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "b149d2a2261b4ff7a7cff85591297106"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "ad706ca03e7bd5ec514eb79431b900d3"
server_key_iv: "820de03830dcd7023548fc67595ad782"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.479904 2446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "b149d2a2261b4ff7a7cff85591297106"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "ad706ca03e7bd5ec514eb79431b900d3"
server_key_iv: "820de03830dcd7023548fc67595ad782"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.481417 2446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:55.482352 2460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.482630 2446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:55.482754 2446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "b149d2a2261b4ff7a7cff85591297106"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "ad706ca03e7bd5ec514eb79431b900d3"
server_key_iv: "820de03830dcd7023548fc67595ad782"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.482837 2446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.495393 2446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.495759 2446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.495919 2446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.496181 2446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:55.496615 2446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:55.496678 2446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.496723 2446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:55.496748 2446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.502858 2446 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:34259
I20260926 09:22:55.502899 2573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:34259 every 8 connection(s)
I20260926 09:22:55.503264 2446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:55.509284 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.504116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60489 (local address 127.24.235.126:40539)
0926 09:22:55.504204 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.504206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.504532 (+ 326us) server_negotiation.cc:407] Connection header received
0926 09:22:55.505368 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.505370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.505382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.505444 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.506136 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.506648 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.507636 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.507859 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.507935 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.508920 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.508921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.508921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.508927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.508948 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.508952 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.509007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.509195 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.509219 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:22:55.509510 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.504108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:40539 (local address 127.24.235.66:60489)
0926 09:22:55.504352 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:55.504401 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:55.505248 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:55.505510 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:55.505513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:55.505522 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:55.506029 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.506033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.506738 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.506739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.507540 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.507542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.507973 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.507974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.508140 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.508760 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:55.508761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:55.508790 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:55.508813 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:55.509101 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:55.509107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:55.509110 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:55.509184 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:22:55.509271 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
I20260926 09:22:55.510039 2574 heartbeater.cc:347] Connected to a master server at 127.24.235.126:40539
I20260926 09:22:55.510154 2574 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:55.510337 2574 heartbeater.cc:511] Master 127.24.235.126:40539 requested a full tablet report, sending...
I20260926 09:22:55.510815 2255 ts_manager.cc:194] Registered new tserver with Master: b149d2a2261b4ff7a7cff85591297106 (127.24.235.66:34259)
I20260926 09:22:55.511181 2255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.66:60489
I20260926 09:22:55.513120 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2446
I20260926 09:22:55.513201 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:55.513357 25517 external_mini_cluster.cc:1428] Setting key 875a468a1451ffc67b649dbe1b932af9
I20260926 09:22:55.514241 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--builtin_ntp_servers=127.24.235.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 09:22:55.600315 2577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.600521 2577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.600543 2577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.601666 2577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.601702 2577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.601727 2577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:55.602988 2577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:40539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.603178 2577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.603431 2577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:55.608565 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.608664 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.608625 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.608783 2577 server_base.cc:1034] running on GCE node
I20260926 09:22:55.609164 2577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.609413 2577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.610572 2577 hybrid_clock.cc:648] HybridClock initialized: now 1790414575610562 us; error 30 us; skew 500 ppm
I20260926 09:22:55.612030 2577 webserver.cc:460] Webserver started at http://127.24.235.67:34643/ using document root <none> and password file <none>
I20260926 09:22:55.612294 2577 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.612363 2577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.612524 2577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.613718 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "e1858370c5bf4aa4be9009dae1c60137"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "1596530518a93d2441559dd818aba0c5"
server_key_iv: "767f44f3dbe4777d7fb1aab6c6636a4f"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.614069 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "e1858370c5bf4aa4be9009dae1c60137"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "1596530518a93d2441559dd818aba0c5"
server_key_iv: "767f44f3dbe4777d7fb1aab6c6636a4f"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.615356 2577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:55.616097 2591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.616297 2577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:55.616400 2577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "e1858370c5bf4aa4be9009dae1c60137"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "1596530518a93d2441559dd818aba0c5"
server_key_iv: "767f44f3dbe4777d7fb1aab6c6636a4f"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.616473 2577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.628510 2577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.628830 2577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.628976 2577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.629262 2577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:55.629609 2577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:55.629642 2577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.629671 2577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:55.629683 2577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.635794 2577 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:43575
I20260926 09:22:55.635890 2704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:43575 every 8 connection(s)
I20260926 09:22:55.636241 2577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:55.639597 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2577
I20260926 09:22:55.639748 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:55.639932 25517 external_mini_cluster.cc:1428] Setting key 3fbc792f3283170e6b7fb7f232818aef
I20260926 09:22:55.642210 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.637140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:57437 (local address 127.24.235.126:40539)
0926 09:22:55.637189 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.637191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.637499 (+ 308us) server_negotiation.cc:407] Connection header received
0926 09:22:55.638069 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.638071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.638084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.638142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.638883 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.639385 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.640643 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.640841 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.640918 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.641863 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.641864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.641865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.641870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.641885 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.641889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.641960 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.642101 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.642131 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:22:55.642458 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.637193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:40539 (local address 127.24.235.67:57437)
0926 09:22:55.637377 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:55.637413 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:55.637939 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:55.638218 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:55.638221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:55.638231 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:55.638772 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.638776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.639483 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.639484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.640570 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:55.640573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:55.640926 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:55.640928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:55.641126 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:55.641715 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:55.641715 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:55.641745 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:55.641768 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:55.641987 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:55.641993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:55.641996 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:55.642089 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 09:22:55.642194 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260926 09:22:55.643010 2705 heartbeater.cc:347] Connected to a master server at 127.24.235.126:40539
I20260926 09:22:55.643121 2705 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:55.643303 2705 heartbeater.cc:511] Master 127.24.235.126:40539 requested a full tablet report, sending...
I20260926 09:22:55.643769 2255 ts_manager.cc:194] Registered new tserver with Master: e1858370c5bf4aa4be9009dae1c60137 (127.24.235.67:43575)
I20260926 09:22:55.644153 2255 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.67:57437
I20260926 09:22:55.652275 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:55.654807 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2315
W20260926 09:22:55.654845 2295 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:55.653995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.24.235.126:40539)
0926 09:22:55.654066 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.654068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.654250 (+ 182us) server_negotiation.cc:407] Connection header received
0926 09:22:55.654324 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.654324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.654338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.654393 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.654744 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39718: BlockingRecv error: recv got EOF from 127.0.0.1:39718 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:55.660910 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2446
I20260926 09:22:55.665693 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2577
I20260926 09:22:55.671265 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2224
2026-09-26T09:22:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (656 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T09:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:55Z Disabled control of system clock
I20260926 09:22:55.699514 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37835
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:22:55.782248 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.782485 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.782509 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.783664 2716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:55.783705 2716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.783718 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.783731 2716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:55.783743 2716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:55.785128 2716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.785349 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.785614 2716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:55.790958 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.790987 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.791050 2716 server_base.cc:1034] running on GCE node
W20260926 09:22:55.791010 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.791497 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.791774 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.792936 2716 hybrid_clock.cc:648] HybridClock initialized: now 1790414575792920 us; error 36 us; skew 500 ppm
I20260926 09:22:55.794325 2716 webserver.cc:460] Webserver started at http://127.24.235.126:33361/ using document root <none> and password file <none>
I20260926 09:22:55.794598 2716 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.794701 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.794872 2716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.795990 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "b69da20bb1bf45f9949d2534e87c6244"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "649a5c9e40e7b61dafa1a5b35158d382"
server_key_iv: "0ab15d7b8dce344c4515b1c5545e9668"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.796356 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "b69da20bb1bf45f9949d2534e87c6244"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "649a5c9e40e7b61dafa1a5b35158d382"
server_key_iv: "0ab15d7b8dce344c4515b1c5545e9668"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.797932 2716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:55.798835 2730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.799047 2716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:55.799141 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "b69da20bb1bf45f9949d2534e87c6244"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "649a5c9e40e7b61dafa1a5b35158d382"
server_key_iv: "0ab15d7b8dce344c4515b1c5545e9668"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.799218 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.802798 2716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.803081 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.803223 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.807415 2716 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37835
I20260926 09:22:55.807446 2782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37835 every 8 connection(s)
I20260926 09:22:55.807850 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:55.808533 2783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:55.811434 2783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244: Bootstrap starting.
I20260926 09:22:55.812038 2783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:55.812359 2783 log.cc:824] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:55.813202 2783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244: No bootstrap required, opened a new log
I20260926 09:22:55.814157 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2716
I20260926 09:22:55.814306 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:55.814548 25517 external_mini_cluster.cc:1428] Setting key 4eb076b46acd9c37858b8f997b72f9a8
I20260926 09:22:55.815176 2783 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } }
I20260926 09:22:55.815348 2783 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:55.815393 2783 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69da20bb1bf45f9949d2534e87c6244, State: Initialized, Role: FOLLOWER
I20260926 09:22:55.815528 2783 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } }
I20260926 09:22:55.815604 2783 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:55.815634 2783 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:55.815671 2783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:55.816387 2783 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } }
I20260926 09:22:55.816483 2783 leader_election.cc:302] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69da20bb1bf45f9949d2534e87c6244; no voters:
I20260926 09:22:55.816663 2783 leader_election.cc:288] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:55.816785 2788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:55.816943 2788 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [term 1 LEADER]: Becoming Leader. State: Replica: b69da20bb1bf45f9949d2534e87c6244, State: Running, Role: LEADER
I20260926 09:22:55.816951 2783 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:55.817098 2788 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } }
I20260926 09:22:55.817260 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.815169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.24.235.126:37835)
0926 09:22:55.815437 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.815446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.815480 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:22:55.815597 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.815605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.815665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.815814 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.816650 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.816655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.816655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.816660 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.816706 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.816721 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.816796 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.816966 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.817057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":115,"threads_started":1}
I20260926 09:22:55.817457 2788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69da20bb1bf45f9949d2534e87c6244. Latest consensus state: current_term: 1 leader_uuid: "b69da20bb1bf45f9949d2534e87c6244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } } }
I20260926 09:22:55.817550 2788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:55.817467 2790 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69da20bb1bf45f9949d2534e87c6244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69da20bb1bf45f9949d2534e87c6244" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37835 } } }
I20260926 09:22:55.818163 2790 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244 [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:55.818375 2792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:55.818919 2792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:55.820453 2792 catalog_manager.cc:1347] Generated new cluster ID: 4097df1f3ac44413b44110c46d8a83a7
I20260926 09:22:55.820508 2792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:55.836997 2792 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:55.837248 2792 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:55.846566 2792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69da20bb1bf45f9949d2534e87c6244: Generated new TSK 0
I20260926 09:22:55.846848 2792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:22:55.854393 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--builtin_ntp_servers=127.24.235.84:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 09:22:55.938942 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:55.939191 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:55.939230 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:55.940402 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:55.940469 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:55.940511 2807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:55.941895 2807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:55.942214 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:55.942526 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:55.948745 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.948757 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:55.948755 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:55.949075 2807 server_base.cc:1034] running on GCE node
I20260926 09:22:55.949370 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:55.949640 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:55.950807 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790414575950788 us; error 38 us; skew 500 ppm
I20260926 09:22:55.952178 2807 webserver.cc:460] Webserver started at http://127.24.235.65:44193/ using document root <none> and password file <none>
I20260926 09:22:55.952443 2807 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:55.952513 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:55.952678 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:55.953851 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "525566b199554cc1a34caa9545b2084a"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "f976b0d29504ee019136bcec2df208aa"
server_key_iv: "903dbb1335951b20105327843a2bb17d"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.954233 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "525566b199554cc1a34caa9545b2084a"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "f976b0d29504ee019136bcec2df208aa"
server_key_iv: "903dbb1335951b20105327843a2bb17d"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.955561 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:22:55.956372 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.956575 2807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:55.956702 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "525566b199554cc1a34caa9545b2084a"
format_stamp: "Formatted at 2026-09-26 09:22:55 on dist-test-slave-7zlw"
server_key: "f976b0d29504ee019136bcec2df208aa"
server_key_iv: "903dbb1335951b20105327843a2bb17d"
server_key_version: "encryptionkey@0"
I20260926 09:22:55.956779 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:55.960415 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:55.960726 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:55.960877 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:55.961138 2807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:55.961485 2807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:55.961542 2807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.961583 2807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:55.961609 2807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:55.967839 2807 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46333
I20260926 09:22:55.967912 2934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46333 every 8 connection(s)
I20260926 09:22:55.968315 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:22:55.969937 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2807
I20260926 09:22:55.970047 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:55.970260 25517 external_mini_cluster.cc:1428] Setting key d35c9af8bf2ec42bbb1c96c607d82280
I20260926 09:22:55.971424 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--builtin_ntp_servers=127.24.235.84:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 09:22:55.971974 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.969113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:50229 (local address 127.24.235.126:37835)
0926 09:22:55.969186 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:55.969189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:55.969451 (+ 262us) server_negotiation.cc:407] Connection header received
0926 09:22:55.970085 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:55.970087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:55.970096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:55.970182 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:55.971708 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:55.971710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:55.971710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:55.971715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:55.971731 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:55.971736 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:55.971768 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:55.971902 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 09:22:55.971920 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:22:55.972224 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:55.969109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37835 (local address 127.24.235.65:50229)
0926 09:22:55.969319 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:55.969354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:55.969938 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:55.970268 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:55.970273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:55.970284 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:55.971517 (+ 1233us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:55.971518 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:55.971562 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:55.971599 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:55.971810 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:55.971815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:55.971818 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:55.971895 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:22:55.971964 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260926 09:22:55.972921 2935 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37835
I20260926 09:22:55.973060 2935 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:55.973209 2935 heartbeater.cc:511] Master 127.24.235.126:37835 requested a full tablet report, sending...
I20260926 09:22:55.973726 2747 ts_manager.cc:194] Registered new tserver with Master: 525566b199554cc1a34caa9545b2084a (127.24.235.65:46333)
W20260926 09:22:56.055560 2938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:56.055828 2938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:56.055871 2938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:56.057137 2938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:56.057206 2938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:56.057251 2938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:56.058635 2938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:56.058925 2938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:56.059228 2938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:56.065608 2943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:56.065539 2946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:56.065629 2938 server_base.cc:1034] running on GCE node
W20260926 09:22:56.065647 2944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:56.066045 2938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:56.066345 2938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:56.067562 2938 hybrid_clock.cc:648] HybridClock initialized: now 1790414576067556 us; error 57 us; skew 500 ppm
I20260926 09:22:56.068938 2938 webserver.cc:460] Webserver started at http://127.24.235.66:36425/ using document root <none> and password file <none>
I20260926 09:22:56.069217 2938 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:56.069293 2938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:56.069453 2938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:56.070566 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "53834dcb1d8e4e27b256faa0b3183eb2"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "31b9debcce940bb1aad0cf0cf5e99dd7"
server_key_iv: "bda7e9b5bd3f45c2b239276c0b917857"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.070971 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "53834dcb1d8e4e27b256faa0b3183eb2"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "31b9debcce940bb1aad0cf0cf5e99dd7"
server_key_iv: "bda7e9b5bd3f45c2b239276c0b917857"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.072453 2938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:56.073446 2952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.073684 2938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:22:56.073801 2938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "53834dcb1d8e4e27b256faa0b3183eb2"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "31b9debcce940bb1aad0cf0cf5e99dd7"
server_key_iv: "bda7e9b5bd3f45c2b239276c0b917857"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.073890 2938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:56.077543 2938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:56.077813 2938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:56.077968 2938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:56.078191 2938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:56.078532 2938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:56.078593 2938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.078637 2938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:56.078668 2938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.084921 2938 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:37381
I20260926 09:22:56.085078 3065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:37381 every 8 connection(s)
I20260926 09:22:56.085418 2938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:22:56.086891 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 2938
I20260926 09:22:56.086994 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:56.087195 25517 external_mini_cluster.cc:1428] Setting key 1b93f496e4be219b80fae526dfc3b7fd
I20260926 09:22:56.088420 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--builtin_ntp_servers=127.24.235.84:46017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 09:22:56.089314 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:56.086228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:57897 (local address 127.24.235.126:37835)
0926 09:22:56.086291 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:56.086293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:56.086531 (+ 238us) server_negotiation.cc:407] Connection header received
0926 09:22:56.087331 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:56.087332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:56.087340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:56.087372 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:56.088989 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:56.088990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:56.088990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:56.088994 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:56.089058 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:56.089063 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:56.089135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:56.089247 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:22:56.089262 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:22:56.089519 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:56.086229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37835 (local address 127.24.235.66:57897)
0926 09:22:56.086414 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:56.086446 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:56.087191 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:56.087540 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:56.087544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:56.087551 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:56.088786 (+ 1235us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:56.088786 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:56.088828 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:56.088863 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:56.089139 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:56.089144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:56.089147 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:56.089223 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:22:56.089284 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260926 09:22:56.089990 3066 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37835
I20260926 09:22:56.090082 3066 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:56.090240 3066 heartbeater.cc:511] Master 127.24.235.126:37835 requested a full tablet report, sending...
I20260926 09:22:56.090574 2747 ts_manager.cc:194] Registered new tserver with Master: 53834dcb1d8e4e27b256faa0b3183eb2 (127.24.235.66:37381)
W20260926 09:22:56.171836 3069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:56.172050 3069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:56.172073 3069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:56.173346 3069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:56.173390 3069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:56.173417 3069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:56.174898 3069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:56.175109 3069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:56.175359 3069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:56.180998 3077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:56.181047 3074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:56.181290 3069 server_base.cc:1034] running on GCE node
W20260926 09:22:56.181375 3075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:56.181607 3069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:56.181875 3069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:56.183075 3069 hybrid_clock.cc:648] HybridClock initialized: now 1790414576183060 us; error 26 us; skew 500 ppm
I20260926 09:22:56.184592 3069 webserver.cc:460] Webserver started at http://127.24.235.67:34053/ using document root <none> and password file <none>
I20260926 09:22:56.184841 3069 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:56.184917 3069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:56.185110 3069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:56.186264 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "7ea49a9730bd417aa66c51709bb09a46"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "7063badc20f3a18a682cdca69515c40c"
server_key_iv: "29dd9d3d877ee0fe9402669fe1d9f1be"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.186635 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "7ea49a9730bd417aa66c51709bb09a46"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "7063badc20f3a18a682cdca69515c40c"
server_key_iv: "29dd9d3d877ee0fe9402669fe1d9f1be"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.188233 3069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:56.188948 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.189162 3069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:56.189268 3069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "7ea49a9730bd417aa66c51709bb09a46"
format_stamp: "Formatted at 2026-09-26 09:22:56 on dist-test-slave-7zlw"
server_key: "7063badc20f3a18a682cdca69515c40c"
server_key_iv: "29dd9d3d877ee0fe9402669fe1d9f1be"
server_key_version: "encryptionkey@0"
I20260926 09:22:56.189342 3069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:56.193198 3069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:56.193461 3069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:56.193614 3069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:56.193842 3069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:56.194161 3069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:56.194214 3069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.194254 3069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:56.194281 3069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:56.200052 3069 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:38405
I20260926 09:22:56.200085 3196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:38405 every 8 connection(s)
I20260926 09:22:56.200415 3069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:56.203800 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:56.201211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:48081 (local address 127.24.235.126:37835)
0926 09:22:56.201259 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:56.201262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:56.201551 (+ 289us) server_negotiation.cc:407] Connection header received
0926 09:22:56.202099 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:56.202101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:56.202106 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:56.202167 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:56.203586 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:56.203588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:56.203588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:22:56.203593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:56.203608 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:22:56.203621 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:56.203680 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:56.203731 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:22:56.203744 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:22:56.204047 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:56.201213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37835 (local address 127.24.235.67:48081)
0926 09:22:56.201405 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:56.201440 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:56.201973 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:56.202318 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:56.202323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:56.202334 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:56.203371 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:22:56.203371 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:56.203401 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:22:56.203430 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:56.203676 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:56.203680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:56.203682 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:22:56.203738 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:22:56.203789 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260926 09:22:56.204180 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 3069
I20260926 09:22:56.204267 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:56.204497 25517 external_mini_cluster.cc:1428] Setting key 5a4990f60ad98ba04206f68cbf3fee26
I20260926 09:22:56.204531 3197 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37835
I20260926 09:22:56.204640 3197 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:56.204833 3197 heartbeater.cc:511] Master 127.24.235.126:37835 requested a full tablet report, sending...
I20260926 09:22:56.205289 2746 ts_manager.cc:194] Registered new tserver with Master: 7ea49a9730bd417aa66c51709bb09a46 (127.24.235.67:38405)
I20260926 09:22:56.216851 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 09:22:56.219235 2787 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:22:56.218612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.24.235.126:37835)
0926 09:22:56.218685 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:56.218688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:56.218884 (+ 196us) server_negotiation.cc:407] Connection header received
0926 09:22:56.218916 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:56.218917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:56.218922 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:56.218982 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:56.219165 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45944: BlockingRecv error: recv got EOF from 127.0.0.1:45944 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:56.219303 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2807
I20260926 09:22:56.224514 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2938
I20260926 09:22:56.229903 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 3069
I20260926 09:22:56.234525 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 2716
2026-09-26T09:22:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (563 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:56 dist-test-slave-7zlw krb5kdc[3207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:56 dist-test-slave-7zlw krb5kdc[3207](info): set up 2 sockets
Sep 26 09:22:56 dist-test-slave-7zlw krb5kdc[3207](info): commencing operation
krb5kdc: starting...
W20260926 09:22:57.290186 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:22:57.416775 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33187
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:57.499469 3220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:57.499687 3220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:57.499709 3220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:57.500864 3220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:22:57.500910 3220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:57.500924 3220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:57.500937 3220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:22:57.500949 3220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:22:57.502456 3220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:57.502710 3220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:57.502969 3220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:57.507877 3225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:57.507972 3226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:57.508042 3220 server_base.cc:1034] running on GCE node
W20260926 09:22:57.507963 3228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:57.508373 3220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:57.508668 3220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:57.509810 3220 hybrid_clock.cc:648] HybridClock initialized: now 1790414577509811 us; error 39 us; skew 500 ppm
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:57.518066 3220 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:22:57.518566 3220 webserver.cc:460] Webserver started at http://127.24.235.126:41539/ using document root <none> and password file <none>
I20260926 09:22:57.518826 3220 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:57.518895 3220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:57.519044 3220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:57.520152 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "9e901b8c5cf34af59e9ff69c44a8459a"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "a85d9023af0a90cbf7ffb694436bd841"
server_key_iv: "f37a3600045d3050c28f56da86337f6d"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.520540 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "9e901b8c5cf34af59e9ff69c44a8459a"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "a85d9023af0a90cbf7ffb694436bd841"
server_key_iv: "f37a3600045d3050c28f56da86337f6d"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.521965 3220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:57.522735 3235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.522905 3220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:22:57.523013 3220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "9e901b8c5cf34af59e9ff69c44a8459a"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "a85d9023af0a90cbf7ffb694436bd841"
server_key_iv: "f37a3600045d3050c28f56da86337f6d"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.523090 3220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:57.534547 3220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:57.536393 3220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:57.536556 3220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:57.540499 3220 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33187
I20260926 09:22:57.540534 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33187 every 8 connection(s)
I20260926 09:22:57.540880 3220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:22:57.541857 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 3220
I20260926 09:22:57.541965 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:22:57.541966 3288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:57.542212 25517 external_mini_cluster.cc:1428] Setting key 8277ba098520bae1ddd59cbe6941f26b
I20260926 09:22:57.545228 3288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: Bootstrap starting.
I20260926 09:22:57.545847 3288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:57.546129 3288 log.cc:824] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:57.546804 3288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: No bootstrap required, opened a new log
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:57.549099 3288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } }
I20260926 09:22:57.549232 3288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:57.549252 3288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e901b8c5cf34af59e9ff69c44a8459a, State: Initialized, Role: FOLLOWER
I20260926 09:22:57.549389 3288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } }
I20260926 09:22:57.549489 3288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:22:57.549536 3288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:22:57.549587 3288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:57.550292 3288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } }
I20260926 09:22:57.550418 3288 leader_election.cc:302] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e901b8c5cf34af59e9ff69c44a8459a; no voters:
I20260926 09:22:57.550662 3288 leader_election.cc:288] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:22:57.550700 3293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:57.550904 3293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [term 1 LEADER]: Becoming Leader. State: Replica: 9e901b8c5cf34af59e9ff69c44a8459a, State: Running, Role: LEADER
I20260926 09:22:57.550938 3288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:22:57.551694 3293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } }
I20260926 09:22:57.552623 3294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e901b8c5cf34af59e9ff69c44a8459a. Latest consensus state: current_term: 1 leader_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } } }
I20260926 09:22:57.552712 3294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [sys.catalog]: This master's current role is: LEADER
I20260926 09:22:57.552549 3295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e901b8c5cf34af59e9ff69c44a8459a" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33187 } } }
I20260926 09:22:57.552914 3295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a [sys.catalog]: This master's current role is: LEADER
W20260926 09:22:57.553174 3307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:22:57.553234 3307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:22:57.553404 3308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:22:57.553550 3308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:22:57.553622 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:57.542737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.24.235.126:33187)
0926 09:22:57.542900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:57.542908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:57.542933 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:22:57.543437 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:57.543443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:57.543494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:57.543645 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:57.544366 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.545387 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.546260 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.546671 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.546791 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:57.549482 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:57.549486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:57.549487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:57.549499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:57.551595 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.552010 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.552013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.552014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.552130 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.552342 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.552343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.552343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.552634 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:57.553191 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:57.553266 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:22:57.553417 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260926 09:22:57.554978 3308 catalog_manager.cc:1347] Generated new cluster ID: e19f5fd6238247f698e37ed7eae7525f
I20260926 09:22:57.555028 3308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:22:57.568266 3308 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:22:57.568922 3308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:22:57.578814 3308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: Generated new TSK 0
I20260926 09:22:57.579061 3308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:22:57.613212 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:57.704195 3314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:57.704455 3314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:57.704509 3314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:57.705909 3314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:57.705986 3314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:57.706036 3314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:22:57.707666 3314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:57.707978 3314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:57.708251 3314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:57.713737 3322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:57.713804 3319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:57.713737 3320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:57.714061 3314 server_base.cc:1034] running on GCE node
I20260926 09:22:57.714285 3314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:57.714485 3314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:57.715641 3314 hybrid_clock.cc:648] HybridClock initialized: now 1790414577715643 us; error 46 us; skew 500 ppm
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:57.718168 3314 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:22:57.718640 3314 webserver.cc:460] Webserver started at http://127.24.235.65:35395/ using document root <none> and password file <none>
I20260926 09:22:57.718878 3314 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:57.718946 3314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:57.719099 3314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:57.720160 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "b981db9b5b4d4303b373e4f77d14f8cd"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "4ab1737d0bcd2a5c0178303fb02f4652"
server_key_iv: "03d1a76fd62b45bd4c0a09469857e8c1"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.720534 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "b981db9b5b4d4303b373e4f77d14f8cd"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "4ab1737d0bcd2a5c0178303fb02f4652"
server_key_iv: "03d1a76fd62b45bd4c0a09469857e8c1"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.721925 3314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:22:57.722656 3329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.722852 3314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.722955 3314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "b981db9b5b4d4303b373e4f77d14f8cd"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "4ab1737d0bcd2a5c0178303fb02f4652"
server_key_iv: "03d1a76fd62b45bd4c0a09469857e8c1"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.723030 3314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:57.730772 3314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:57.733155 3314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:57.733315 3314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:57.733558 3314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:57.733877 3314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:57.733932 3314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.733971 3314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:57.733999 3314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.739965 3314 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:39951
I20260926 09:22:57.740005 3442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:39951 every 8 connection(s)
I20260926 09:22:57.740322 3314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:57.749253 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:57.741125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:37303 (local address 127.24.235.126:33187)
0926 09:22:57.741191 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:57.741193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:57.741394 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:22:57.741894 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:57.741894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:57.741903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:57.741961 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:57.742854 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.743339 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.744312 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.744506 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.744557 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:57.746680 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:57.746681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:57.746681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:57.746687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:57.748079 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.748420 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.748421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.748422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.748470 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.748673 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.748674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.748674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.748772 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:57.748821 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:57.749124 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 09:22:57.749187 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:57.749457 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:57.741095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33187 (local address 127.24.235.65:37303)
0926 09:22:57.741278 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:57.741319 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:57.741812 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:57.742015 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:57.742019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:57.742584 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:57.742781 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:57.742785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.743419 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.743420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:57.744237 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:57.744239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.744577 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.744577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:57.744703 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:57.745232 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:57.745240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:57.746594 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:57.748146 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:57.748151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:57.748153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:57.748359 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:57.748526 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:57.748526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:57.748527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:57.748622 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:57.748826 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:57.748830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:57.748973 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 09:22:57.749126 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 09:22:57.749246 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 09:22:57.749890 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 3314
I20260926 09:22:57.749991 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:22:57.749972 3443 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33187
I20260926 09:22:57.750061 3443 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:57.750208 25517 external_mini_cluster.cc:1428] Setting key 609b595721e700762b521a159a056c78
I20260926 09:22:57.750231 3443 heartbeater.cc:511] Master 127.24.235.126:33187 requested a full tablet report, sending...
I20260926 09:22:57.750816 3251 ts_manager.cc:194] Registered new tserver with Master: b981db9b5b4d4303b373e4f77d14f8cd (127.24.235.65:39951)
I20260926 09:22:57.751448 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:37303
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:22:57.776630 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:57.859616 3448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:57.859829 3448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:57.859854 3448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:57.860930 3448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:57.860963 3448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:57.860991 3448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:22:57.862502 3448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:57.862690 3448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:57.862870 3448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:57.867882 3453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:57.867955 3448 server_base.cc:1034] running on GCE node
W20260926 09:22:57.867916 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:57.867913 3454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:57.868430 3448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:57.868731 3448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:57.869896 3448 hybrid_clock.cc:648] HybridClock initialized: now 1790414577869889 us; error 57 us; skew 500 ppm
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:57.872382 3448 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:22:57.872865 3448 webserver.cc:460] Webserver started at http://127.24.235.66:38763/ using document root <none> and password file <none>
I20260926 09:22:57.873150 3448 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:57.873221 3448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:57.873387 3448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:57.874476 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "3d457fb69797494999e209b852ce8b54"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "ddfb398bfb2886372ab3d79f87f5b2c3"
server_key_iv: "6499f029dd4154787347cb1268080a63"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.874856 3448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "3d457fb69797494999e209b852ce8b54"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "ddfb398bfb2886372ab3d79f87f5b2c3"
server_key_iv: "6499f029dd4154787347cb1268080a63"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.876261 3448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:22:57.877210 3463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.877408 3448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:22:57.877509 3448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "3d457fb69797494999e209b852ce8b54"
format_stamp: "Formatted at 2026-09-26 09:22:57 on dist-test-slave-7zlw"
server_key: "ddfb398bfb2886372ab3d79f87f5b2c3"
server_key_iv: "6499f029dd4154787347cb1268080a63"
server_key_version: "encryptionkey@0"
I20260926 09:22:57.877600 3448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:57.886299 3448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:57.888331 3448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:57.888507 3448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:57.888759 3448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:57.889145 3448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:57.889202 3448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.889245 3448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:57.889271 3448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:57.895289 3448 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:40801
I20260926 09:22:57.895329 3576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:40801 every 8 connection(s)
I20260926 09:22:57.895674 3448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:22:57 dist-test-slave-7zlw krb5kdc[3207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414577, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:57.901995 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 3448
I20260926 09:22:57.902144 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:22:57.902356 25517 external_mini_cluster.cc:1428] Setting key f7d113a1d102ac1d0099fdb5addf98e9
I20260926 09:22:57.905776 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:57.896459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:53179 (local address 127.24.235.126:33187)
0926 09:22:57.896527 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:57.896530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:57.896741 (+ 211us) server_negotiation.cc:407] Connection header received
0926 09:22:57.897370 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:57.897371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:57.897384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:57.897436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:57.898404 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.898919 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.899936 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.900138 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.900223 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:57.902449 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:57.902451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:57.902451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:57.902459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:57.904055 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.904507 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.904509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.904509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.904580 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:57.904856 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:57.904857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:57.904858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:57.905010 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:57.905174 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:57.905549 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 09:22:57.905709 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:22:57.905987 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:57.896454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33187 (local address 127.24.235.66:53179)
0926 09:22:57.896616 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:57.896652 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:57.897255 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:57.897509 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:57.897513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:57.898096 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:57.898299 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:57.898304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.899039 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.899040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:57.899833 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:57.899835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:57.900234 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:57.900235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:57.900354 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:57.900859 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:57.900868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:57.902356 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:57.904146 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:57.904152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:57.904155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:57.904429 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:57.904664 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:57.904664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:57.904665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:57.904769 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:57.905169 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:57.905177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:57.905365 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 09:22:57.905520 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 09:22:57.905710 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260926 09:22:57.906433 3577 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33187
I20260926 09:22:57.906533 3577 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:57.906726 3577 heartbeater.cc:511] Master 127.24.235.126:33187 requested a full tablet report, sending...
I20260926 09:22:57.907116 3251 ts_manager.cc:194] Registered new tserver with Master: 3d457fb69797494999e209b852ce8b54 (127.24.235.66:40801)
I20260926 09:22:57.907498 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:53179
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:22:57.932717 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:22:58.013622 3582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:22:58.013877 3582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:22:58.013914 3582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:22:58.015113 3582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:22:58.015188 3582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:22:58.015234 3582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:22:58.016696 3582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:22:58.016969 3582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:22:58.017225 3582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:22:58.022341 3590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:58.022337 3587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:22:58.022337 3588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:22:58.022523 3582 server_base.cc:1034] running on GCE node
I20260926 09:22:58.022765 3582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:22:58.023051 3582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:22:58.024268 3582 hybrid_clock.cc:648] HybridClock initialized: now 1790414578024255 us; error 43 us; skew 500 ppm
Sep 26 09:22:58 dist-test-slave-7zlw krb5kdc[3207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414578, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:22:58.027405 3582 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:22:58.027889 3582 webserver.cc:460] Webserver started at http://127.24.235.67:34581/ using document root <none> and password file <none>
I20260926 09:22:58.028142 3582 fs_manager.cc:359] Metadata directory not provided
I20260926 09:22:58.028229 3582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:22:58.028385 3582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:22:58.029522 3582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "2730648702344b0b9192d2232e5129e3"
format_stamp: "Formatted at 2026-09-26 09:22:58 on dist-test-slave-7zlw"
server_key: "b558ed77921a57274dfee96c8a548389"
server_key_iv: "42e6b944e3c49b6a8f9522a0e6f699bc"
server_key_version: "encryptionkey@0"
I20260926 09:22:58.029875 3582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "2730648702344b0b9192d2232e5129e3"
format_stamp: "Formatted at 2026-09-26 09:22:58 on dist-test-slave-7zlw"
server_key: "b558ed77921a57274dfee96c8a548389"
server_key_iv: "42e6b944e3c49b6a8f9522a0e6f699bc"
server_key_version: "encryptionkey@0"
I20260926 09:22:58.031176 3582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 09:22:58.031944 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:58.032140 3582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:22:58.032245 3582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "2730648702344b0b9192d2232e5129e3"
format_stamp: "Formatted at 2026-09-26 09:22:58 on dist-test-slave-7zlw"
server_key: "b558ed77921a57274dfee96c8a548389"
server_key_iv: "42e6b944e3c49b6a8f9522a0e6f699bc"
server_key_version: "encryptionkey@0"
I20260926 09:22:58.032327 3582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:22:58.045244 3582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:22:58.047564 3582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:22:58.047765 3582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:22:58.048036 3582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:22:58.048412 3582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:22:58.048485 3582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:58.048547 3582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:22:58.048612 3582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:22:58.056130 3582 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:44993
I20260926 09:22:58.056169 3710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:44993 every 8 connection(s)
I20260926 09:22:58.056560 3582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:22:58.057891 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 3582
I20260926 09:22:58.057967 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:22:58.058166 25517 external_mini_cluster.cc:1428] Setting key 9f72c75db8307d0d67d4c346a07ea9a3
Sep 26 09:22:58 dist-test-slave-7zlw krb5kdc[3207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414578, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:22:58.066268 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.057418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:45917 (local address 127.24.235.126:33187)
0926 09:22:58.057475 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.057477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.057692 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:22:58.058307 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.058308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.058324 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.058395 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:58.059413 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.059924 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.061053 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.061294 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.061397 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.063603 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:58.063604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:58.063604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:58.063613 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:58.064990 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:58.065460 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:58.065461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:58.065461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:58.065514 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:58.065729 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:58.065729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:58.065729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:58.065833 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:58.065884 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.066127 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.066200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:22:58.066419 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.057345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33187 (local address 127.24.235.67:45917)
0926 09:22:58.057539 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.057586 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.058181 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.058478 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.058483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.059086 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:22:58.059291 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.059296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.060060 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.060061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.060934 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.060936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.061409 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.061410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.061562 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.062070 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:22:58.062083 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:22:58.063497 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:22:58.065157 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:58.065164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:58.065165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:58.065393 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:58.065573 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:22:58.065573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:22:58.065574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:22:58.065675 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:22:58.065887 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:22:58.065891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:22:58.066007 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.066089 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.066213 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260926 09:22:58.066931 3711 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33187
I20260926 09:22:58.067044 3711 heartbeater.cc:464] Registering TS with master...
I20260926 09:22:58.067231 3711 heartbeater.cc:511] Master 127.24.235.126:33187 requested a full tablet report, sending...
I20260926 09:22:58.067641 3251 ts_manager.cc:194] Registered new tserver with Master: 2730648702344b0b9192d2232e5129e3 (127.24.235.67:44993)
I20260926 09:22:58.068039 3251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:45917
I20260926 09:22:58.069849 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:22:58.077759 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.071187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.24.235.126:33187)
0926 09:22:58.071268 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.071270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.071343 (+ 73us) server_negotiation.cc:407] Connection header received
0926 09:22:58.071420 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.071421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.071432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.071487 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:22:58.072098 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.072592 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.073405 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.073633 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.073721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.075003 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:22:58.075004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:22:58.075005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:22:58.075012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:22:58.076447 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:58.076846 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:58.076847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:58.076848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:58.076911 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:22:58.077175 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:22:58.077176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:22:58.077176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:22:58.077340 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:22:58.077453 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.077564 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.077653 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:22:58.079558 3251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:22:58.081490 3251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:22:58.088254 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.083947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:44993 (local address 127.0.0.1:53016)
0926 09:22:58.084465 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.084475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.084523 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.084756 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.084757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.084759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:58.084797 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.084798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.086107 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.086108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.087057 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.087059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.087941 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.087942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.088022 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.088036 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.088109 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.088157 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":127,"thread_start_us":82,"threads_started":1}
I20260926 09:22:58.088294 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.084054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.24.235.67:44993)
0926 09:22:58.084273 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.084279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.084523 (+ 244us) server_negotiation.cc:407] Connection header received
0926 09:22:58.084591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.084593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.084608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.084679 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:58.084878 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.085979 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.087189 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.087857 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.087933 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.087988 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.088138 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.088181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":121,"threads_started":1}
I20260926 09:22:58.088795 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.083935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:39951 (local address 127.0.0.1:51788)
0926 09:22:58.084284 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.084330 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.084397 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.084647 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.084650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.084652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:58.084728 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.084730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.086121 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.086121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.087637 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.087639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.088527 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.088528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.088587 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.088591 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.088651 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.088704 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":65,"thread_start_us":56,"threads_started":1}
I20260926 09:22:58.089059 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.084282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.24.235.65:39951)
0926 09:22:58.084480 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.084484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.084496 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:22:58.084540 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.084542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.084559 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.084643 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:58.084816 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.086024 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.087725 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.088444 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.088517 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.088572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.088913 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.088956 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260926 09:22:58.089080 3645 tablet_service.cc:1467] Processing CreateTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 (DEFAULT_TABLE table=test-table [id=cfe044442eb04181904016df6ae436e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:58.089480 3645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b0bbd4e4474818bdd74bbe35502482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:58.089612 3377 tablet_service.cc:1467] Processing CreateTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 (DEFAULT_TABLE table=test-table [id=cfe044442eb04181904016df6ae436e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:58.089846 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b0bbd4e4474818bdd74bbe35502482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:58.092214 3728 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: Bootstrap starting.
I20260926 09:22:58.092458 3729 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Bootstrap starting.
I20260926 09:22:58.092651 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.084282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.24.235.66:40801)
0926 09:22:58.084808 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.084812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.084821 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:22:58.084858 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.084860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.084874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.084947 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:58.085216 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.085897 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.087491 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.088234 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.088301 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.092464 (+ 4163us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.092499 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.092562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":120,"threads_started":1}
I20260926 09:22:58.092855 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.084131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40801 (local address 127.0.0.1:42034)
0926 09:22:58.084497 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.084509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.084563 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.085057 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.085058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.085060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:58.085133 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.085135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.086001 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.086002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.087403 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.087406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.088682 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.088683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.088741 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.088745 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.088785 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.088824 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260926 09:22:58.092900 3728 tablet_bootstrap.cc:654] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:58.093140 3729 tablet_bootstrap.cc:654] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:58.093495 3728 log.cc:824] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:58.093642 3511 tablet_service.cc:1467] Processing CreateTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 (DEFAULT_TABLE table=test-table [id=cfe044442eb04181904016df6ae436e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:22:58.093760 3729 log.cc:824] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:58.093981 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b0bbd4e4474818bdd74bbe35502482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:22:58.094417 3728 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: No bootstrap required, opened a new log
I20260926 09:22:58.094497 3729 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: No bootstrap required, opened a new log
I20260926 09:22:58.094497 3728 ts_tablet_manager.cc:1397] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:58.094555 3729 ts_tablet_manager.cc:1397] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:22:58.096509 3732 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Bootstrap starting.
I20260926 09:22:58.096495 3729 raft_consensus.cc:348] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.096494 3728 raft_consensus.cc:348] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.096673 3729 raft_consensus.cc:374] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:58.096683 3728 raft_consensus.cc:374] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:58.096722 3729 raft_consensus.cc:729] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2730648702344b0b9192d2232e5129e3, State: Initialized, Role: FOLLOWER
I20260926 09:22:58.096722 3728 raft_consensus.cc:729] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b981db9b5b4d4303b373e4f77d14f8cd, State: Initialized, Role: FOLLOWER
I20260926 09:22:58.096843 3728 consensus_queue.cc:260] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.096892 3729 consensus_queue.cc:260] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.097169 3728 ts_tablet_manager.cc:1428] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 09:22:58.097407 3732 tablet_bootstrap.cc:654] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Neither blocks nor log segments found. Creating new log.
I20260926 09:22:58.097625 3443 heartbeater.cc:503] Master 127.24.235.126:33187 was elected leader, sending a full tablet report...
I20260926 09:22:58.098014 3732 log.cc:824] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Log is configured to *not* fsync() on all Append() calls
I20260926 09:22:58.098264 3711 heartbeater.cc:503] Master 127.24.235.126:33187 was elected leader, sending a full tablet report...
I20260926 09:22:58.098289 3729 ts_tablet_manager.cc:1428] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 09:22:58.099129 3732 tablet_bootstrap.cc:492] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: No bootstrap required, opened a new log
I20260926 09:22:58.099215 3732 ts_tablet_manager.cc:1397] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:22:58.100582 3732 raft_consensus.cc:348] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.100746 3732 raft_consensus.cc:374] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:22:58.100793 3732 raft_consensus.cc:729] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d457fb69797494999e209b852ce8b54, State: Initialized, Role: FOLLOWER
I20260926 09:22:58.100911 3732 consensus_queue.cc:260] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.101217 3732 ts_tablet_manager.cc:1428] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:22:58.101289 3577 heartbeater.cc:503] Master 127.24.235.126:33187 was elected leader, sending a full tablet report...
W20260926 09:22:58.146281 3578 tablet.cc:2369] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:22:58.153448 3734 raft_consensus.cc:482] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:22:58.153586 3734 raft_consensus.cc:504] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.153967 3734 leader_election.cc:288] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b981db9b5b4d4303b373e4f77d14f8cd (127.24.235.65:39951), 3d457fb69797494999e209b852ce8b54 (127.24.235.66:40801)
I20260926 09:22:58.157088 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.154379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:49057 (local address 127.24.235.65:39951)
0926 09:22:58.154425 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.154427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.154437 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:22:58.154467 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.154468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.154482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.154544 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:58.154721 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.155357 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156313 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.156707 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156797 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.156806 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.156961 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.156998 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:22:58.157102 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.154112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:39951 (local address 127.24.235.67:49057)
0926 09:22:58.154184 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.154198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.154251 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.154563 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.154564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.154566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:58.154639 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.154641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.155440 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.155441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.156213 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.156215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.156793 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.156883 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.156886 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.156956 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.156998 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:22:58.157125 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.154244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40801 (local address 127.24.235.67:40979)
0926 09:22:58.154468 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.154478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:22:58.154543 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:22:58.154817 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:22:58.154818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:22:58.154820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:22:58.154885 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.154886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.155491 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.155492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.156237 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:22:58.156239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.156830 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:22:58.156903 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.156912 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 09:22:58.156971 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:22:58.157011 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":169,"threads_started":1}
I20260926 09:22:58.157125 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.154340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:40979 (local address 127.24.235.66:40801)
0926 09:22:58.154441 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.154444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.154711 (+ 267us) server_negotiation.cc:407] Connection header received
0926 09:22:58.154734 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.154734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.154745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.154784 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:22:58.154958 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.155400 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156336 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.156731 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.156820 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.156828 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.156990 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.157054 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 09:22:58.157522 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b0bbd4e4474818bdd74bbe35502482" candidate_uuid: "2730648702344b0b9192d2232e5129e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" is_pre_election: true
I20260926 09:22:58.157518 3531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b0bbd4e4474818bdd74bbe35502482" candidate_uuid: "2730648702344b0b9192d2232e5129e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d457fb69797494999e209b852ce8b54" is_pre_election: true
I20260926 09:22:58.157703 3397 raft_consensus.cc:2454] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2730648702344b0b9192d2232e5129e3 in term 0.
I20260926 09:22:58.157718 3531 raft_consensus.cc:2454] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2730648702344b0b9192d2232e5129e3 in term 0.
I20260926 09:22:58.157922 3598 leader_election.cc:302] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2730648702344b0b9192d2232e5129e3, b981db9b5b4d4303b373e4f77d14f8cd; no voters:
I20260926 09:22:58.158054 3734 raft_consensus.cc:2781] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:22:58.158150 3734 raft_consensus.cc:482] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:22:58.158186 3734 raft_consensus.cc:3037] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:58.158916 3734 raft_consensus.cc:504] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.159067 3734 leader_election.cc:288] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [CANDIDATE]: Term 1 election: Requested vote from peers b981db9b5b4d4303b373e4f77d14f8cd (127.24.235.65:39951), 3d457fb69797494999e209b852ce8b54 (127.24.235.66:40801)
I20260926 09:22:58.159307 3531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b0bbd4e4474818bdd74bbe35502482" candidate_uuid: "2730648702344b0b9192d2232e5129e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d457fb69797494999e209b852ce8b54"
I20260926 09:22:58.159318 3397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b0bbd4e4474818bdd74bbe35502482" candidate_uuid: "2730648702344b0b9192d2232e5129e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b981db9b5b4d4303b373e4f77d14f8cd"
I20260926 09:22:58.159394 3531 raft_consensus.cc:3037] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:58.159392 3397 raft_consensus.cc:3037] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:22:58.160178 3531 raft_consensus.cc:2454] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2730648702344b0b9192d2232e5129e3 in term 1.
I20260926 09:22:58.160180 3397 raft_consensus.cc:2454] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2730648702344b0b9192d2232e5129e3 in term 1.
I20260926 09:22:58.160353 3598 leader_election.cc:302] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2730648702344b0b9192d2232e5129e3, 3d457fb69797494999e209b852ce8b54; no voters:
I20260926 09:22:58.160470 3734 raft_consensus.cc:2781] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:22:58.160681 3734 raft_consensus.cc:686] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 1 LEADER]: Becoming Leader. State: Replica: 2730648702344b0b9192d2232e5129e3, State: Running, Role: LEADER
I20260926 09:22:58.160802 3734 consensus_queue.cc:237] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } }
I20260926 09:22:58.161741 3251 catalog_manager.cc:5495] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2730648702344b0b9192d2232e5129e3 (127.24.235.67). New cstate: current_term: 1 leader_uuid: "2730648702344b0b9192d2232e5129e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2730648702344b0b9192d2232e5129e3" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 44993 } health_report { overall_health: HEALTHY } } }
I20260926 09:22:58.183763 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.181419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.24.235.67:44993)
0926 09:22:58.181490 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:22:58.181492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:22:58.181513 (+ 21us) server_negotiation.cc:407] Connection header received
0926 09:22:58.181555 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:22:58.181556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:22:58.181573 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:22:58.181630 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:22:58.181770 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.182254 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.182989 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:22:58.183186 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:22:58.183261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:22:58.183392 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:22:58.183490 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:22:58.183552 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:22:58.183587 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 09:22:58.183610 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:22:58.185662 3531 raft_consensus.cc:1261] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 1 FOLLOWER]: Refusing update from remote peer 2730648702344b0b9192d2232e5129e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:58.185662 3397 raft_consensus.cc:1261] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 1 FOLLOWER]: Refusing update from remote peer 2730648702344b0b9192d2232e5129e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:22:58.186010 3734 consensus_queue.cc:1035] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b981db9b5b4d4303b373e4f77d14f8cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:58.186179 3738 consensus_queue.cc:1035] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d457fb69797494999e209b852ce8b54" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:22:58.188817 3743 mvcc.cc:205] Tried to move back new op lower bound from 7333538112245465088 to 7333538112147562496. Current Snapshot: MvccSnapshot[applied={T|T < 7333538112245465088}]
I20260926 09:22:58.188941 3742 mvcc.cc:205] Tried to move back new op lower bound from 7333538112245465088 to 7333538112147562496. Current Snapshot: MvccSnapshot[applied={T|T < 7333538112245465088}]
I20260926 09:22:58.191293 3250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:58.191406 3250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:22:58.192319 3250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e901b8c5cf34af59e9ff69c44a8459a: Sending DeleteTablet for 3 replicas of tablet 67b0bbd4e4474818bdd74bbe35502482
I20260926 09:22:58.192708 3511 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:58 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:42034
I20260926 09:22:58.192730 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:58 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:51788
I20260926 09:22:58.192727 3645 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b0bbd4e4474818bdd74bbe35502482 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:22:58 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:53016
I20260926 09:22:58.192951 3751 tablet_replica.cc:330] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: stopping tablet replica
I20260926 09:22:58.193053 3752 tablet_replica.cc:330] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd: stopping tablet replica
I20260926 09:22:58.193097 3753 tablet_replica.cc:330] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: stopping tablet replica
I20260926 09:22:58.193148 3752 raft_consensus.cc:2229] T 67b0bbd4e4474818bdd74bbe35502482 P b981db9b5b4d4303b373e4f77d14f8cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:58.193172 3753 raft_consensus.cc:2229] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:22:58.193181 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 3314
I20260926 09:22:58.197778 3751 raft_consensus.cc:2229] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:22:58.197932 3751 raft_consensus.cc:2258] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:58.198046 3753 raft_consensus.cc:2258] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:22:58.198556 3753 ts_tablet_manager.cc:1905] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:22:58.198583 3751 ts_tablet_manager.cc:1905] T 67b0bbd4e4474818bdd74bbe35502482 P 3d457fb69797494999e209b852ce8b54: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:22:58.198798 3236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:39951 (error 108)
I20260926 09:22:58.199257 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 3448
I20260926 09:22:58.202845 3753 ts_tablet_manager.cc:1918] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:22:58.202944 3753 log.cc:1196] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/67b0bbd4e4474818bdd74bbe35502482
I20260926 09:22:58.203253 3753 ts_tablet_manager.cc:1939] T 67b0bbd4e4474818bdd74bbe35502482 P 2730648702344b0b9192d2232e5129e3: Deleting consensus metadata
I20260926 09:22:58.203552 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.203387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:39951 (local address 127.0.0.1:51804)
0926 09:22:58.203445 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.203501 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:22:58.203694 3236 catalog_manager.cc:4584] TS b981db9b5b4d4303b373e4f77d14f8cd (127.24.235.65:39951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67b0bbd4e4474818bdd74bbe35502482: Network error: Client connection negotiation failed: client connection to 127.24.235.65:39951: connect: Connection refused (error 111)
I20260926 09:22:58.204093 3239 catalog_manager.cc:4841] TS 2730648702344b0b9192d2232e5129e3 (127.24.235.67:44993): tablet 67b0bbd4e4474818bdd74bbe35502482 (table test-table [id=cfe044442eb04181904016df6ae436e4]) successfully deleted
I20260926 09:22:58.205430 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:22:58.205304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40801 (local address 127.0.0.1:42044)
0926 09:22:58.205363 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:22:58.205396 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.66:40801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:22:58.205637 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 3582
I20260926 09:22:58.211452 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 3220
2026-09-26T09:22:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1977 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:22:58 dist-test-slave-7zlw krb5kdc[3757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:22:58 dist-test-slave-7zlw krb5kdc[3757](info): set up 2 sockets
Sep 26 09:22:58 dist-test-slave-7zlw krb5kdc[3757](info): commencing operation
krb5kdc: starting...
W20260926 09:22:59.262243 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:22:59 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:22:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:22:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangeradmin_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangeradmin_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangerlookup_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangerlookup_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
I20260926 09:22:59.445868 25517 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskr9gC17/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/postgres -l logfile start
2026-09-26 09:23:02.436 UTC [3787] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:23:02.436 UTC [3787] LOG: listening on IPv4 address "127.24.235.84", port 48233
2026-09-26 09:23:02.440 UTC [3787] LOG: listening on Unix socket "/tmp/.s.PGSQL.48233"
2026-09-26 09:23:02.477 UTC [3790] LOG: database system was shut down at 2026-09-26 09:23:00 UTC
2026-09-26 09:23:02.482 UTC [3787] LOG: database system is ready to accept connections
I20260926 09:23:03.438112 25517 mini_postgres.cc:96] Postgres bound to 48233
2026-09-26 09:23:03.444 UTC [3798] FATAL: database "slave" does not exist
127.24.235.84:48233 - accepting connections
I20260926 09:23:03.445428 25517 mini_ranger.cc:161] Starting Ranger...
I20260926 09:23:03.461370 25517 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:23:04.070142 25517 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:23:04.070250 25517 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin
2026-09-26 09:23:04,425 [I] DB FLAVOR :POSTGRES
2026-09-26 09:23:04,426 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:23:04,426 [I] Checking connection..
2026-09-26 09:23:04,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:23:04,704 [I] Checking connection passed.
2026-09-26 09:23:04,704 [I] --------- Verifying version history table ---------
2026-09-26 09:23:04,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:04,974 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:23:04,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:05,247 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:23:05,247 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:23:05,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:23:05.501 UTC [3862] WARNING: there is no transaction in progress
2026-09-26 09:23:05,508 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:23:05,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:05,752 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:23:05,752 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:23:05,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:23:06,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:23:06,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zlw.c.gcp-upstream.internal','N') ;"
2026-09-26 09:23:06,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:23:06,767 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:23:06,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:23:07,037 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:23:07,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:23:07,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:23:07,525 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:23:07,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:23:08.560 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.578 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.596 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.615 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.635 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.657 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.721 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.729 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.746 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.757 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.768 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.783 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.794 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.802 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.811 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:08.821 UTC [3970] WARNING: there is no transaction in progress
2026-09-26 09:23:09,607 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:23:09,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:23:09,846 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:23:09,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:23:10,118 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:23:10,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:23:10,371 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:23:10,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:23:10,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:23:10,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zlw.c.gcp-upstream.internal';"
2026-09-26 09:23:11,096 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:23:11,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:23:11,357 [I] DB_PATCHES have already been applied
I20260926 09:23:11.362920 25517 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskr9gC17/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:23:11.362993 25517 mini_ranger.cc:193] Using host: 127.24.235.84
I20260926 09:23:11.366595 25517 mini_ranger.cc:227] Ranger admin URL: http://127.24.235.84:36863
Sep 26, 2026 9:23:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:23:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:23:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:23:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:23:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:23:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:23:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:23:12.529848 25517 mini_ranger.cc:160] Time spent starting Ranger: real 9.084s user 0.004s sys 0.006s
Sep 26, 2026 9:23:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:23:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 9:23:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:23:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:23:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:23:35.399068 25517 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangerkms_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/rangerkms_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.235.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.235.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:23:35 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414615, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:23:35.509949 25517 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:23:35.521046 25517 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:23:36.624768 25517 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:23:36.624854 25517 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms
I20260926 09:23:36.624876 25517 mini_ranger_kms.cc:227] Using postgres at 127.24.235.84:48233
2026-09-26 09:23:36,865 [I] DB FLAVOR :POSTGRES
2026-09-26 09:23:36,865 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:23:36,865 [I] Checking connection
2026-09-26 09:23:36,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:23:37,134 [I] connection success
2026-09-26 09:23:37,134 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:23:37,134 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:23:37,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:23:37,380 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:23:37,381 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:23:37,381 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:23:37,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:48233/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:23:37,661 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:23:38.212384 25517 mini_ranger_kms.cc:310] Created kms service
I20260926 09:23:38.426728 25517 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:23:38.475157 25517 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:23:38 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414615, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:23:38.910388 25517 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:23:38.910512 25517 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms:/tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:23:38.910534 25517 mini_ranger_kms.cc:240] Using host: 127.24.235.84
I20260926 09:23:38.914641 25517 mini_ranger_kms.cc:276] Ranger KMS PID: 4151
I20260926 09:23:38.914808 25517 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.235.84:60671
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:23:41.216281 25517 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.706s user 0.004s sys 0.219s
I20260926 09:23:41.216387 25517 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:23:41.216470 25517 mini_ranger_kms.cc:401] 127.24.235.84:60671/kms/v1/keys
Sep 26 09:23:41 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:23:41 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
Sep 26 09:23:42 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414621, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
Sep 26 09:23:44 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:23:44.757452 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37365
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37365
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:60671/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:23:44.866546 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:23:44.866828 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:23:44.866868 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:23:44.868244 4201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:23:44.868319 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:23:44.868343 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:23:44.868391 4201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:23:44.868439 4201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:23:44.870302 4201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.235.84:60671/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37365
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:23:44.870620 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:23:44.870908 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:23:44.877085 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:44.877117 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:23:44.877085 4201 server_base.cc:1034] running on GCE node
W20260926 09:23:44.877092 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:23:44.877548 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:23:44.877902 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:23:44.879087 4201 hybrid_clock.cc:648] HybridClock initialized: now 1790414624879047 us; error 55 us; skew 500 ppm
Sep 26 09:23:44 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414624, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:23:44.882951 4201 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:23:44.883527 4201 webserver.cc:460] Webserver started at http://127.24.235.126:36653/ using document root <none> and password file <none>
I20260926 09:23:44.883816 4201 fs_manager.cc:359] Metadata directory not provided
I20260926 09:23:44.883870 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:23:44.883999 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:23:44 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414624, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:23:44.935124 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "96a04a86509f41dc970a5771d8e59729"
format_stamp: "Formatted at 2026-09-26 09:23:44 on dist-test-slave-7zlw"
server_key: "9ee757e4d0b686e42bf07580d8ffcd32"
server_key_iv: "b41bbf364fbee794fe39064f4ca825c4"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:44.935561 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "96a04a86509f41dc970a5771d8e59729"
format_stamp: "Formatted at 2026-09-26 09:23:44 on dist-test-slave-7zlw"
server_key: "9ee757e4d0b686e42bf07580d8ffcd32"
server_key_iv: "b41bbf364fbee794fe39064f4ca825c4"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:44.937112 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:23:44.953315 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:44.953542 4201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:23:44.953678 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "96a04a86509f41dc970a5771d8e59729"
format_stamp: "Formatted at 2026-09-26 09:23:44 on dist-test-slave-7zlw"
server_key: "9ee757e4d0b686e42bf07580d8ffcd32"
server_key_iv: "b41bbf364fbee794fe39064f4ca825c4"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:44.953809 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:23:44.966200 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:23:44.968937 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:23:44.969288 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:23:44.977454 4201 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37365
I20260926 09:23:44.977479 4268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37365 every 8 connection(s)
I20260926 09:23:44.978014 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:23:44.979067 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:23:44.983264 4269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: Bootstrap starting.
I20260926 09:23:44.984323 4269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: Neither blocks nor log segments found. Creating new log.
I20260926 09:23:44.984719 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 4201
I20260926 09:23:44.984848 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:23:44.984907 4269 log.cc:824] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: Log is configured to *not* fsync() on all Append() calls
I20260926 09:23:44.985157 25517 external_mini_cluster.cc:1428] Setting key b4cd7dcefa9cacce01da5faaf2d5e718
I20260926 09:23:44.986179 4269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: No bootstrap required, opened a new log
I20260926 09:23:44.989809 4269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } }
I20260926 09:23:44.989995 4269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:23:44.990025 4269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a04a86509f41dc970a5771d8e59729, State: Initialized, Role: FOLLOWER
I20260926 09:23:44.990190 4269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } }
I20260926 09:23:44.990270 4269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:23:44.990298 4269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:23:44.990388 4269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:23:44.991508 4269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } }
I20260926 09:23:44.991626 4269 leader_election.cc:302] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a04a86509f41dc970a5771d8e59729; no voters:
I20260926 09:23:44.991861 4269 leader_election.cc:288] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:23:44.991945 4274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:23:44.992265 4269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:23:44.992332 4274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [term 1 LEADER]: Becoming Leader. State: Replica: 96a04a86509f41dc970a5771d8e59729, State: Running, Role: LEADER
I20260926 09:23:44.992513 4274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } }
Sep 26 09:23:44 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:23:44.994112 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a04a86509f41dc970a5771d8e59729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } } }
I20260926 09:23:44.994202 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [sys.catalog]: This master's current role is: LEADER
I20260926 09:23:44.994346 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a04a86509f41dc970a5771d8e59729. Latest consensus state: current_term: 1 leader_uuid: "96a04a86509f41dc970a5771d8e59729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a04a86509f41dc970a5771d8e59729" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37365 } } }
I20260926 09:23:44.994426 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729 [sys.catalog]: This master's current role is: LEADER
I20260926 09:23:44.997329 4283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:23:44.997920 4283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:23:44.999920 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:44.985796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.24.235.126:37365)
0926 09:23:44.986058 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:44.986066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:44.986139 (+ 73us) server_negotiation.cc:407] Connection header received
0926 09:23:44.986765 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:44.986773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:44.986827 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:44.986991 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:23:44.987677 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:44.988828 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:44.989667 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:44.990322 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:44.990472 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:44.993387 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:23:44.993391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:23:44.993392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:23:44.993403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:23:44.996545 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:44.997098 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:44.997102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:44.997102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:44.997275 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:44.997656 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:44.997658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:44.997658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:44.998186 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:23:44.999135 (+ 949us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:44.999247 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 09:23:44.999468 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
I20260926 09:23:45.002799 4269 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:23:45.013628 4283 catalog_manager.cc:1347] Generated new cluster ID: 3280ad2af34344469462268bc503abdd
I20260926 09:23:45.013738 4283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:23:45.026021 4283 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:23:45.026772 4283 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:23:45.030083 4283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: Generated new TSK 0
I20260926 09:23:45.030262 4283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:23:45.889724 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:60671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:23:46.036896 4314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:23:46.037243 4314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:23:46.037284 4314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:23:46.039211 4314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:23:46.039301 4314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:23:46.039352 4314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:23:46.041576 4314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.235.84:60671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:23:46.041911 4314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:23:46.042191 4314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:23:46.050673 4314 server_base.cc:1034] running on GCE node
W20260926 09:23:46.050791 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.050634 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.050606 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:23:46.051270 4314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:23:46.051647 4314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:23:46.052881 4314 hybrid_clock.cc:648] HybridClock initialized: now 1790414626052860 us; error 61 us; skew 500 ppm
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:23:46.057082 4314 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:23:46.057682 4314 webserver.cc:460] Webserver started at http://127.24.235.65:35481/ using document root <none> and password file <none>
I20260926 09:23:46.058022 4314 fs_manager.cc:359] Metadata directory not provided
I20260926 09:23:46.058203 4314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:23:46.058369 4314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:23:46.074434 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "0c6e1a94f94440fc837919f7d6296b7e"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "5b9bb8fad4b12b0ed38c28032007abde"
server_key_iv: "3247fa1f8bbc2e90558350f0f5b79b75"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.075057 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "0c6e1a94f94440fc837919f7d6296b7e"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "5b9bb8fad4b12b0ed38c28032007abde"
server_key_iv: "3247fa1f8bbc2e90558350f0f5b79b75"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.077450 4314 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 09:23:46.101902 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.102141 4314 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.001s
I20260926 09:23:46.102257 4314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "0c6e1a94f94440fc837919f7d6296b7e"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "5b9bb8fad4b12b0ed38c28032007abde"
server_key_iv: "3247fa1f8bbc2e90558350f0f5b79b75"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.102367 4314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:23:46.121578 4314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:23:46.124034 4314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:23:46.124318 4314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:23:46.124790 4314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:23:46.125362 4314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:23:46.125461 4314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.125566 4314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:23:46.125645 4314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.139232 4314 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:45301
I20260926 09:23:46.139343 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:45301 every 8 connection(s)
I20260926 09:23:46.139662 4314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:23:46.144443 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 4314
I20260926 09:23:46.144644 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:23:46.144862 25517 external_mini_cluster.cc:1428] Setting key 71b192d0fe9b0124f9a602290a2d81f4
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:23:46.153847 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.140503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37365 (local address 127.24.235.65:43923)
0926 09:23:46.140697 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.140742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.141329 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.141679 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.141685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.141983 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:23:46.142400 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.142407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.143176 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.143177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.144678 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.144682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.145453 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.145454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.145615 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.146296 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:23:46.146307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:23:46.148474 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:23:46.151440 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.151450 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.151453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.151854 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.152124 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.152125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.152125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.152305 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.152713 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:23:46.152721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:23:46.152943 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.153232 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.153485 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260926 09:23:46.154508 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.140566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:43923 (local address 127.24.235.126:37365)
0926 09:23:46.141123 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.141125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.141139 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:23:46.141457 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.141459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.141474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.141549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:23:46.142533 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.143062 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.144805 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.145302 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.145373 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.148649 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:23:46.148651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:23:46.148651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:23:46.148661 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:23:46.151302 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.151956 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.151958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.151958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.152036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.152398 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.152399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.152399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.152600 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:23:46.152678 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.154280 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.154400 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":73,"threads_started":1}
I20260926 09:23:46.155475 4445 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37365
I20260926 09:23:46.155596 4445 heartbeater.cc:464] Registering TS with master...
I20260926 09:23:46.155807 4445 heartbeater.cc:511] Master 127.24.235.126:37365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:23:46.156433 4233 ts_manager.cc:194] Registered new tserver with Master: 0c6e1a94f94440fc837919f7d6296b7e (127.24.235.65:45301)
I20260926 09:23:46.157469 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:43923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:23:46.180295 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:60671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 09:23:46.342552 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:23:46.342880 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:23:46.342921 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:23:46.345058 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:23:46.345209 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:23:46.345301 4451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:23:46.348340 4451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.235.84:60671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:23:46.349092 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:23:46.349689 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:23:46.358731 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.359059 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.358843 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:23:46.358964 4451 server_base.cc:1034] running on GCE node
I20260926 09:23:46.359458 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:23:46.359748 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:23:46.360931 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790414626360910 us; error 34 us; skew 500 ppm
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:23:46.365295 4451 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:23:46.365969 4451 webserver.cc:460] Webserver started at http://127.24.235.66:36839/ using document root <none> and password file <none>
I20260926 09:23:46.366266 4451 fs_manager.cc:359] Metadata directory not provided
I20260926 09:23:46.366335 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:23:46.366525 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:23:46.379844 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "491e5fac7d67464c8a5a5986e1c642cd"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "18f433ee506bda86b7315b8ec7619ed1"
server_key_iv: "4995a40293dee4187c72366e31ebfdd7"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.380317 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "491e5fac7d67464c8a5a5986e1c642cd"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "18f433ee506bda86b7315b8ec7619ed1"
server_key_iv: "4995a40293dee4187c72366e31ebfdd7"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.382177 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 09:23:46.405208 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.405413 4451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 09:23:46.405507 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "491e5fac7d67464c8a5a5986e1c642cd"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "18f433ee506bda86b7315b8ec7619ed1"
server_key_iv: "4995a40293dee4187c72366e31ebfdd7"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.405616 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:23:46.420327 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:23:46.423400 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:23:46.423619 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:23:46.423892 4451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:23:46.424523 4451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:23:46.424584 4451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.424654 4451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:23:46.424685 4451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.439371 4451 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:39683
I20260926 09:23:46.439877 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:23:46.440634 4582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:39683 every 8 connection(s)
I20260926 09:23:46.447429 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 4451
I20260926 09:23:46.447664 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:23:46.447894 25517 external_mini_cluster.cc:1428] Setting key 32de19c47a41f0ac9d1b71a4ed4bb4fb
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:23:46.456499 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.441375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:40033 (local address 127.24.235.126:37365)
0926 09:23:46.441426 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.441428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.445382 (+ 3954us) server_negotiation.cc:407] Connection header received
0926 09:23:46.446186 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.446188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.446208 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.446292 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:23:46.447170 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.448001 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.449593 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.449920 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.449962 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.452638 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:23:46.452639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:23:46.452640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:23:46.452649 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:23:46.454669 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.455189 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.455190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.455190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.455249 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.455678 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.455679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.455679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.455851 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:23:46.455911 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.456302 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.456402 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:23:46.457495 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.441378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37365 (local address 127.24.235.66:40033)
0926 09:23:46.445163 (+ 3785us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.445234 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.446025 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.446404 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.446410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.446683 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:23:46.447026 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.447034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.448129 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.448130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.449486 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.449489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.450064 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.450065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.450211 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.450826 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:23:46.450837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:23:46.452521 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:23:46.454775 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.454784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.454787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.455097 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.455319 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.455320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.455320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.455443 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.455966 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:23:46.455973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:23:46.456149 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.456951 (+ 802us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.457209 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3723,"thread_start_us":104,"threads_started":1}
I20260926 09:23:46.458372 4583 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37365
I20260926 09:23:46.458493 4583 heartbeater.cc:464] Registering TS with master...
I20260926 09:23:46.458696 4583 heartbeater.cc:511] Master 127.24.235.126:37365 requested a full tablet report, sending...
I20260926 09:23:46.459156 4233 ts_manager.cc:194] Registered new tserver with Master: 491e5fac7d67464c8a5a5986e1c642cd (127.24.235.66:39683)
I20260926 09:23:46.459689 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:40033
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:23:46.485369 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:60671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:23:46.625602 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:23:46.626014 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:23:46.626122 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:23:46.628008 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:23:46.628077 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:23:46.628122 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:23:46.630389 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.235.84:60671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:23:46.630736 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:23:46.631022 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:23:46.636693 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.637976 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:23:46.638123 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:23:46.639400 4588 server_base.cc:1034] running on GCE node
I20260926 09:23:46.639628 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:23:46.639958 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:23:46.641110 4588 hybrid_clock.cc:648] HybridClock initialized: now 1790414626641087 us; error 41 us; skew 500 ppm
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:23:46.643915 4588 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:23:46.644486 4588 webserver.cc:460] Webserver started at http://127.24.235.67:40555/ using document root <none> and password file <none>
I20260926 09:23:46.644760 4588 fs_manager.cc:359] Metadata directory not provided
I20260926 09:23:46.644829 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:23:46.644979 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:23:46.658902 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "f8e954a449ca4a9a8a1c525bc95eabc4"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "ba443631bde8f0cc96718bb723a53250"
server_key_iv: "e5732be4812e46b31dc75b6c343e175a"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.659483 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e954a449ca4a9a8a1c525bc95eabc4"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "ba443631bde8f0cc96718bb723a53250"
server_key_iv: "e5732be4812e46b31dc75b6c343e175a"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.661737 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 09:23:46.675501 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.675704 4588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 09:23:46.675798 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "f8e954a449ca4a9a8a1c525bc95eabc4"
format_stamp: "Formatted at 2026-09-26 09:23:46 on dist-test-slave-7zlw"
server_key: "ba443631bde8f0cc96718bb723a53250"
server_key_iv: "e5732be4812e46b31dc75b6c343e175a"
server_key_version: "kuduclusterkey@0"
I20260926 09:23:46.675907 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:23:46.693739 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:23:46.696092 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:23:46.696269 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:23:46.696602 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:23:46.697067 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:23:46.697113 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.697145 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:23:46.697172 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:23:46.710598 4588 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:39383
I20260926 09:23:46.711112 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:23:46.716852 4716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:39383 every 8 connection(s)
I20260926 09:23:46.718470 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 4588
I20260926 09:23:46.718585 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:23:46.718801 25517 external_mini_cluster.cc:1428] Setting key 906e1c1b97c2dae6bc5ba19d098f187a
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:23:46.728832 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.718621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:34603 (local address 127.24.235.126:37365)
0926 09:23:46.718682 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.718685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.719296 (+ 611us) server_negotiation.cc:407] Connection header received
0926 09:23:46.719959 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.719960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.719972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.720010 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:23:46.720882 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.721447 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.722663 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.722860 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.722950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.725599 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:23:46.725600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:23:46.725601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:23:46.725608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:23:46.727291 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.727762 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.727763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.727763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.727845 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.728108 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.728109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.728109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.728258 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:23:46.728300 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.728648 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.728732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:23:46.729491 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.718240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37365 (local address 127.24.235.67:34603)
0926 09:23:46.719009 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.719218 (+ 209us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.719845 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.720100 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.720104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.720455 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:23:46.720771 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.720776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.721550 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.721551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.722552 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.722555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.722959 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.722960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.723143 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.723831 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:23:46.723839 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:23:46.725489 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:23:46.727389 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.727395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.727398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.727678 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.727917 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:23:46.727918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:23:46.727918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:23:46.728040 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:23:46.728346 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:23:46.728351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:23:46.728507 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.728991 (+ 484us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.729186 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":110,"threads_started":1}
I20260926 09:23:46.733114 4717 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37365
I20260926 09:23:46.733235 4717 heartbeater.cc:464] Registering TS with master...
I20260926 09:23:46.733444 4717 heartbeater.cc:511] Master 127.24.235.126:37365 requested a full tablet report, sending...
I20260926 09:23:46.733810 4233 ts_manager.cc:194] Registered new tserver with Master: f8e954a449ca4a9a8a1c525bc95eabc4 (127.24.235.67:39383)
I20260926 09:23:46.734292 4233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:34603
I20260926 09:23:46.741101 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:23:46.763865 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.745448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.24.235.126:37365)
0926 09:23:46.745498 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.745501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.753265 (+ 7764us) server_negotiation.cc:407] Connection header received
0926 09:23:46.753388 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.753389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.753401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.753434 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:23:46.754396 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.755185 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.756975 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.757313 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.757361 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.759206 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:23:46.759208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:23:46.759208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:23:46.759216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:23:46.760969 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.762895 (+ 1926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.762896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.762897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.762964 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:23:46.763221 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:23:46.763222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:23:46.763223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:23:46.763361 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:23:46.763441 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.763677 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.763763 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:23:46.766568 4233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:23:46.769186 4233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:23:46.807964 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.798246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:45301 (local address 127.0.0.1:37728)
0926 09:23:46.800478 (+ 2232us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.800490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.800549 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.801053 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.801057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.801060 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:23:46.801144 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.801146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.804482 (+ 3336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.804483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.805940 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.805944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.807578 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.807579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.807673 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.807689 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.807787 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.807858 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2186,"thread_start_us":94,"threads_started":1}
I20260926 09:23:46.808620 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.798474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.24.235.65:45301)
0926 09:23:46.799540 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.799547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.800615 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 09:23:46.800661 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.800664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.800685 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.800762 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:23:46.801242 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.804237 (+ 2995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.806084 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.807399 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.807474 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.807540 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.808434 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.808485 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":58,"threads_started":1}
I20260926 09:23:46.809837 4373 tablet_service.cc:1467] Processing CreateTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f (DEFAULT_TABLE table=test-table [id=f76e0c97bf824af1900fd53219a63d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:23:46.810268 4373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cbc5fa029541d6b639300dbf2ef39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:23:46.811574 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.797208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:39383 (local address 127.0.0.1:52158)
0926 09:23:46.799667 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.799720 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.799797 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.801584 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.801585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.801587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:23:46.801636 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.801638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.807770 (+ 6132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.807771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.808966 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.808970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.811324 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.811326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.811395 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.811400 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.811445 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.811489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2404,"thread_start_us":368,"threads_started":1}
I20260926 09:23:46.813156 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.797551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.24.235.67:39383)
0926 09:23:46.801230 (+ 3679us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.801235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.801249 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:23:46.801317 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.801319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.801335 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.801432 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:23:46.806169 (+ 4737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.807638 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.809184 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.811177 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.811255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.812023 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.812058 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.812102 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3611,"thread_start_us":147,"threads_started":1}
I20260926 09:23:46.815546 4634 tablet_service.cc:1467] Processing CreateTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f (DEFAULT_TABLE table=test-table [id=f76e0c97bf824af1900fd53219a63d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:23:46.815940 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cbc5fa029541d6b639300dbf2ef39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:23:46.817037 4736 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Bootstrap starting.
I20260926 09:23:46.818846 4736 tablet_bootstrap.cc:654] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Neither blocks nor log segments found. Creating new log.
I20260926 09:23:46.819159 4736 log.cc:824] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:23:46.820109 4736 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: No bootstrap required, opened a new log
I20260926 09:23:46.820171 4736 ts_tablet_manager.cc:1397] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:23:46.822899 4736 raft_consensus.cc:348] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.823087 4736 raft_consensus.cc:374] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:23:46.823122 4736 raft_consensus.cc:729] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6e1a94f94440fc837919f7d6296b7e, State: Initialized, Role: FOLLOWER
I20260926 09:23:46.823262 4736 consensus_queue.cc:260] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.823581 4736 ts_tablet_manager.cc:1428] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:23:46.823709 4445 heartbeater.cc:503] Master 127.24.235.126:37365 was elected leader, sending a full tablet report...
I20260926 09:23:46.824747 4738 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Bootstrap starting.
I20260926 09:23:46.825119 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.797223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:39683 (local address 127.0.0.1:33846)
0926 09:23:46.800848 (+ 3625us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:46.800860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:46.800903 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:46.821137 (+ 20234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:46.821138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:46.821142 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:23:46.821206 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.821209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.822571 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.822572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.823719 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:46.823724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.824823 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.824824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:46.824889 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.824894 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:23:46.824947 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:23:46.824992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2740,"thread_start_us":207,"threads_started":1}
I20260926 09:23:46.825498 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:46.797404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.24.235.66:39683)
0926 09:23:46.817315 (+ 19911us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:46.817320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:46.817335 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:23:46.817399 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:46.817401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:46.817428 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:46.817593 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:23:46.821297 (+ 3704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.822452 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.823854 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:46.824633 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:46.824702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:46.824784 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:46.825340 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 09:23:46.825399 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11794,"thread_start_us":145,"threads_started":1}
I20260926 09:23:46.825904 4738 tablet_bootstrap.cc:654] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Neither blocks nor log segments found. Creating new log.
I20260926 09:23:46.826216 4738 log.cc:824] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Log is configured to *not* fsync() on all Append() calls
I20260926 09:23:46.826184 4517 tablet_service.cc:1467] Processing CreateTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f (DEFAULT_TABLE table=test-table [id=f76e0c97bf824af1900fd53219a63d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:23:46.826563 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cbc5fa029541d6b639300dbf2ef39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:23:46.827801 4738 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: No bootstrap required, opened a new log
I20260926 09:23:46.827883 4738 ts_tablet_manager.cc:1397] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:23:46.830863 4738 raft_consensus.cc:348] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.831045 4738 raft_consensus.cc:374] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:23:46.831085 4738 raft_consensus.cc:729] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e954a449ca4a9a8a1c525bc95eabc4, State: Initialized, Role: FOLLOWER
I20260926 09:23:46.831220 4738 consensus_queue.cc:260] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.831496 4738 ts_tablet_manager.cc:1428] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:23:46.831612 4717 heartbeater.cc:503] Master 127.24.235.126:37365 was elected leader, sending a full tablet report...
I20260926 09:23:46.832407 4741 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Bootstrap starting.
I20260926 09:23:46.834750 4741 tablet_bootstrap.cc:654] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Neither blocks nor log segments found. Creating new log.
I20260926 09:23:46.836117 4741 log.cc:824] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:23:46.837199 4741 tablet_bootstrap.cc:492] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: No bootstrap required, opened a new log
I20260926 09:23:46.837386 4741 ts_tablet_manager.cc:1397] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 09:23:46.840058 4741 raft_consensus.cc:348] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.840281 4741 raft_consensus.cc:374] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:23:46.840323 4741 raft_consensus.cc:729] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491e5fac7d67464c8a5a5986e1c642cd, State: Initialized, Role: FOLLOWER
I20260926 09:23:46.840480 4741 consensus_queue.cc:260] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:46.840817 4741 ts_tablet_manager.cc:1428] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:23:46.840943 4583 heartbeater.cc:503] Master 127.24.235.126:37365 was elected leader, sending a full tablet report...
W20260926 09:23:46.890749 4446 tablet.cc:2369] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 09:23:46 dist-test-slave-7zlw krb5kdc[3757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414626, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
W20260926 09:23:46.941541 4584 tablet.cc:2369] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 09:23:46.969129 4718 tablet.cc:2369] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:23:47.116948 4739 raft_consensus.cc:482] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:23:47.117174 4739 raft_consensus.cc:504] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:47.117602 4739 leader_election.cc:288] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491e5fac7d67464c8a5a5986e1c642cd (127.24.235.66:39683), f8e954a449ca4a9a8a1c525bc95eabc4 (127.24.235.67:39383)
I20260926 09:23:47.122834 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.117948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:33077 (local address 127.24.235.66:39683)
0926 09:23:47.118428 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:47.118431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:47.118920 (+ 489us) server_negotiation.cc:407] Connection header received
0926 09:23:47.119036 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:47.119037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:47.119064 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:47.119096 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:23:47.119460 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.120186 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.121870 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.122426 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.122478 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:47.122488 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:47.122716 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 09:23:47.122757 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449}
I20260926 09:23:47.123113 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.118040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:39683 (local address 127.24.235.65:33077)
0926 09:23:47.118837 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:47.118849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:47.118952 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:47.119171 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:47.119172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:47.119174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:23:47.119347 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:47.119350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.120293 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.120294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:47.121757 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:47.121760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.122552 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.122552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:47.122627 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:47.122631 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:23:47.122971 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 09:23:47.123016 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":248,"threads_started":1}
I20260926 09:23:47.123514 4537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cbc5fa029541d6b639300dbf2ef39f" candidate_uuid: "0c6e1a94f94440fc837919f7d6296b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e5fac7d67464c8a5a5986e1c642cd" is_pre_election: true
I20260926 09:23:47.123693 4537 raft_consensus.cc:2454] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6e1a94f94440fc837919f7d6296b7e in term 0.
I20260926 09:23:47.123699 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.118687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:41457 (local address 127.24.235.67:39383)
0926 09:23:47.118741 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:47.118743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:47.119153 (+ 410us) server_negotiation.cc:407] Connection header received
0926 09:23:47.119257 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:47.119259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:47.119268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:47.119297 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:23:47.119759 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.120538 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.122334 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.122938 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.123014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:47.123022 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:47.123588 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 09:23:47.123635 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:23:47.123934 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.118796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:39383 (local address 127.24.235.65:41457)
0926 09:23:47.119079 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:47.119090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:23:47.119182 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:23:47.119359 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:23:47.119360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:23:47.119361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:23:47.119672 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:47.119675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.120757 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.120758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:47.122216 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:23:47.122219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.123422 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.123423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:23:47.123501 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:47.123504 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:23:47.123794 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 09:23:47.123839 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":160,"threads_started":1}
I20260926 09:23:47.123960 4333 leader_election.cc:302] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6e1a94f94440fc837919f7d6296b7e, 491e5fac7d67464c8a5a5986e1c642cd; no voters:
I20260926 09:23:47.124145 4739 raft_consensus.cc:2781] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:23:47.124207 4739 raft_consensus.cc:482] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:23:47.124235 4739 raft_consensus.cc:3037] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:23:47.124976 4739 raft_consensus.cc:504] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:47.125164 4739 leader_election.cc:288] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [CANDIDATE]: Term 1 election: Requested vote from peers 491e5fac7d67464c8a5a5986e1c642cd (127.24.235.66:39683), f8e954a449ca4a9a8a1c525bc95eabc4 (127.24.235.67:39383)
I20260926 09:23:47.125336 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cbc5fa029541d6b639300dbf2ef39f" candidate_uuid: "0c6e1a94f94440fc837919f7d6296b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" is_pre_election: true
I20260926 09:23:47.125499 4671 raft_consensus.cc:2454] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c6e1a94f94440fc837919f7d6296b7e in term 0.
I20260926 09:23:47.125823 4671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cbc5fa029541d6b639300dbf2ef39f" candidate_uuid: "0c6e1a94f94440fc837919f7d6296b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4"
I20260926 09:23:47.125900 4671 raft_consensus.cc:3037] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:23:47.126595 4671 raft_consensus.cc:2454] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6e1a94f94440fc837919f7d6296b7e in term 1.
I20260926 09:23:47.126809 4334 leader_election.cc:302] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6e1a94f94440fc837919f7d6296b7e, f8e954a449ca4a9a8a1c525bc95eabc4; no voters:
I20260926 09:23:47.126812 4537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cbc5fa029541d6b639300dbf2ef39f" candidate_uuid: "0c6e1a94f94440fc837919f7d6296b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491e5fac7d67464c8a5a5986e1c642cd"
I20260926 09:23:47.126889 4537 raft_consensus.cc:3037] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:23:47.127579 4537 raft_consensus.cc:2454] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c6e1a94f94440fc837919f7d6296b7e in term 1.
I20260926 09:23:47.127784 4739 raft_consensus.cc:2781] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:23:47.127960 4739 raft_consensus.cc:686] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [term 1 LEADER]: Becoming Leader. State: Replica: 0c6e1a94f94440fc837919f7d6296b7e, State: Running, Role: LEADER
I20260926 09:23:47.128077 4739 consensus_queue.cc:237] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } }
I20260926 09:23:47.128911 4233 catalog_manager.cc:5495] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c6e1a94f94440fc837919f7d6296b7e (127.24.235.65). New cstate: current_term: 1 leader_uuid: "0c6e1a94f94440fc837919f7d6296b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c6e1a94f94440fc837919f7d6296b7e" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45301 } health_report { overall_health: HEALTHY } } }
I20260926 09:23:47.156474 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.153834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.24.235.65:45301)
0926 09:23:47.153879 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:23:47.153882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:23:47.153891 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:23:47.153959 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:23:47.153960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:23:47.153984 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:23:47.154104 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:23:47.154244 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.154718 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.155506 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:23:47.155769 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:23:47.155865 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:23:47.156057 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:23:47.156233 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:23:47.156310 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:23:47.156350 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:23:47.156374 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:23:47.158591 4671 raft_consensus.cc:1261] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 1 FOLLOWER]: Refusing update from remote peer 0c6e1a94f94440fc837919f7d6296b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:23:47.158957 4537 raft_consensus.cc:1261] T 46cbc5fa029541d6b639300dbf2ef39f P 491e5fac7d67464c8a5a5986e1c642cd [term 1 FOLLOWER]: Refusing update from remote peer 0c6e1a94f94440fc837919f7d6296b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:23:47.159484 4747 consensus_queue.cc:1035] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e954a449ca4a9a8a1c525bc95eabc4" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:23:47.159683 4739 consensus_queue.cc:1035] T 46cbc5fa029541d6b639300dbf2ef39f P 0c6e1a94f94440fc837919f7d6296b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "491e5fac7d67464c8a5a5986e1c642cd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:23:47.163132 4752 mvcc.cc:205] Tried to move back new op lower bound from 7333538312838672384 to 7333538312717303808. Current Snapshot: MvccSnapshot[applied={T|T < 7333538312838672384}]
I20260926 09:23:47.165073 4233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:23:47.165162 4233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:23:47.166013 4233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96a04a86509f41dc970a5771d8e59729: Sending DeleteTablet for 3 replicas of tablet 46cbc5fa029541d6b639300dbf2ef39f
I20260926 09:23:47.166378 4517 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:23:47 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:33846
I20260926 09:23:47.166473 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:23:47 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:52158
I20260926 09:23:47.166600 4373 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cbc5fa029541d6b639300dbf2ef39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:23:47 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:37728
I20260926 09:23:47.167309 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 4314
I20260926 09:23:47.168705 4759 tablet_replica.cc:330] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4: stopping tablet replica
I20260926 09:23:47.168830 4759 raft_consensus.cc:2229] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:23:47.168901 4759 raft_consensus.cc:2258] T 46cbc5fa029541d6b639300dbf2ef39f P f8e954a449ca4a9a8a1c525bc95eabc4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:23:47.172799 4470 connection.cc:535] server connection from 127.24.235.65:33077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 09:23:47.172796 4219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:45301 (error 108)
I20260926 09:23:47.173902 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.173732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:45301 (local address 127.0.0.1:37744)
0926 09:23:47.173818 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:47.173863 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:45301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 09:23:47.173898 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 4451
W20260926 09:23:47.174027 4219 catalog_manager.cc:4584] TS 0c6e1a94f94440fc837919f7d6296b7e (127.24.235.65:45301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46cbc5fa029541d6b639300dbf2ef39f: Network error: Client connection negotiation failed: client connection to 127.24.235.65:45301: connect: Connection refused (error 111)
I20260926 09:23:47.187024 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:23:47.183763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:39683 (local address 127.0.0.1:33860)
0926 09:23:47.186906 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0926 09:23:47.186958 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.66:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3112}
I20260926 09:23:47.187362 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 4588
I20260926 09:23:47.196565 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 4201
I20260926 09:23:47.203202 25517 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:23:47.548427 25517 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:23:47.548563 25517 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:23:47.893712 25517 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:23:47.893 UTC [3787] LOG: received smart shutdown request
2026-09-26 09:23:47.898 UTC [3787] LOG: background worker "logical replication launcher" (PID 3796) exited with exit code 1
2026-09-26 09:23:47.898 UTC [3791] LOG: shutting down
2026-09-26 09:23:48.127 UTC [3787] LOG: database system is shut down
2026-09-26T09:23:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50039 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:23:48 dist-test-slave-7zlw krb5kdc[4770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:23:48 dist-test-slave-7zlw krb5kdc[4770](info): set up 2 sockets
Sep 26 09:23:48 dist-test-slave-7zlw krb5kdc[4770](info): commencing operation
krb5kdc: starting...
W20260926 09:23:49.317920 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:23:49 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:23:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangeradmin_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangeradmin_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangerlookup_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangerlookup_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
I20260926 09:23:49.505151 25517 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskr9gC17/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/postgres -l logfile start
2026-09-26 09:23:52.568 UTC [4800] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 09:23:52.568 UTC [4800] LOG: listening on IPv4 address "127.24.235.84", port 53915
2026-09-26 09:23:52.573 UTC [4800] LOG: listening on Unix socket "/tmp/.s.PGSQL.53915"
2026-09-26 09:23:52.610 UTC [4803] LOG: database system was shut down at 2026-09-26 09:23:50 UTC
2026-09-26 09:23:52.614 UTC [4800] LOG: database system is ready to accept connections
I20260926 09:23:53.577163 25517 mini_postgres.cc:96] Postgres bound to 53915
2026-09-26 09:23:53.584 UTC [4811] FATAL: database "slave" does not exist
127.24.235.84:53915 - accepting connections
I20260926 09:23:53.585359 25517 mini_ranger.cc:161] Starting Ranger...
I20260926 09:23:53.605876 25517 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 09:23:54.205461 25517 mini_ranger.cc:87] Created ranger Postgres database
I20260926 09:23:54.205534 25517 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin
2026-09-26 09:23:54,489 [I] DB FLAVOR :POSTGRES
2026-09-26 09:23:54,490 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:23:54,490 [I] Checking connection..
2026-09-26 09:23:54,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 09:23:54,752 [I] Checking connection passed.
2026-09-26 09:23:54,752 [I] --------- Verifying version history table ---------
2026-09-26 09:23:54,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:55,035 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:23:55,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:55,278 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 09:23:55,279 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 09:23:55,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 09:23:55.565 UTC [4875] WARNING: there is no transaction in progress
2026-09-26 09:23:55,572 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 09:23:55,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 09:23:55,812 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 09:23:55,813 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:23:55,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 09:23:56,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 09:23:56,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zlw.c.gcp-upstream.internal','N') ;"
2026-09-26 09:23:56,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:23:56,847 [I] Table x_portal_user does not exist in database ranger
2026-09-26 09:23:56,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:23:57,102 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 09:23:57,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:23:57,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:23:57,594 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 09:23:57,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 09:23:58.614 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.632 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.652 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.670 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.689 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.709 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.771 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.780 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.797 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.809 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.820 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.836 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.847 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.855 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.862 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:58.870 UTC [4983] WARNING: there is no transaction in progress
2026-09-26 09:23:59,639 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 09:23:59,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 09:23:59,877 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 09:23:59,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 09:24:00,127 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 09:24:00,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 09:24:00,406 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 09:24:00,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 09:24:00,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 09:24:00,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zlw.c.gcp-upstream.internal';"
2026-09-26 09:24:01,145 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 09:24:01,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 09:24:01,403 [I] DB_PATCHES have already been applied
I20260926 09:24:01.408869 25517 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskr9gC17/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 09:24:01.408946 25517 mini_ranger.cc:193] Using host: 127.24.235.84
I20260926 09:24:01.412374 25517 mini_ranger.cc:227] Ranger admin URL: http://127.24.235.84:51457
Sep 26, 2026 9:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 9:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 9:24:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 9:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 9:24:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 09:24:02.542392 25517 mini_ranger.cc:160] Time spent starting Ranger: real 8.957s user 0.003s sys 0.007s
Sep 26, 2026 9:24:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 9:24:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 9:24:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 9:24:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 9:24:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 09:24:26.188974 25517 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangerkms_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/rangerkms_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.235.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.235.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.235.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/HTTP_127.24.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 09:24:26 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414666, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 09:24:26.279480 25517 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 09:24:26.289140 25517 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 09:24:27.285079 25517 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 09:24:27.285178 25517 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms
I20260926 09:24:27.285198 25517 mini_ranger_kms.cc:227] Using postgres at 127.24.235.84:53915
2026-09-26 09:24:27,458 [I] DB FLAVOR :POSTGRES
2026-09-26 09:24:27,459 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 09:24:27,459 [I] Checking connection
2026-09-26 09:24:27,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 09:24:27,694 [I] connection success
2026-09-26 09:24:27,694 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 09:24:27,694 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 09:24:27,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 09:24:27,940 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 09:24:27,941 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 09:24:27,941 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 09:24:27,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.235.84:53915/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 09:24:28,226 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 09:24:28.756764 25517 mini_ranger_kms.cc:310] Created kms service
I20260926 09:24:28.958849 25517 mini_ranger_kms.cc:326] Created kudu user
I20260926 09:24:29.005146 25517 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 09:24:29 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414666, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:29.417837 25517 mini_ranger_kms.cc:384] Added ranger policy
I20260926 09:24:29.417954 25517 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms:/tmp/dist-test-taskr9gC17/build/release/bin/postgresql.jar:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskr9gC17/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 09:24:29.417973 25517 mini_ranger_kms.cc:240] Using host: 127.24.235.84
I20260926 09:24:29.422449 25517 mini_ranger_kms.cc:276] Ranger KMS PID: 5164
I20260926 09:24:29.422556 25517 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.235.84:49507
Sep 26, 2026 9:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 9:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 9:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 9:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 9:24:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 9:24:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 09:24:31.732693 25517 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.453s user 0.010s sys 0.146s
I20260926 09:24:31.732803 25517 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 09:24:31.732851 25517 mini_ranger_kms.cc:401] 127.24.235.84:49507/kms/v1/keys
Sep 26 09:24:31 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414671, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 09:24:32 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414671, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
Sep 26 09:24:32 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414671, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.235.84@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
Sep 26 09:24:35 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:24:35.213220 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36261
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36261
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:35.301957 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:35.302229 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:35.302270 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:35.303506 5214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:35.303576 5214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:35.303610 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:35.303656 5214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:35.303684 5214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:35.305363 5214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.235.84:49507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36261
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:35.305651 5214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:35.305888 5214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:35.311580 5219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:35.311683 5222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:35.311589 5220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:35.311772 5214 server_base.cc:1034] running on GCE node
I20260926 09:24:35.312178 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:35.312526 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:35.313726 5214 hybrid_clock.cc:648] HybridClock initialized: now 1790414675313701 us; error 41 us; skew 500 ppm
Sep 26 09:24:35 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414675, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:35.316416 5214 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:35.317013 5214 webserver.cc:460] Webserver started at http://127.24.235.126:37039/ using document root <none> and password file <none>
I20260926 09:24:35.317293 5214 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:35.317358 5214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:35.317507 5214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:24:35 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414675, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:35.370877 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "b0ca701a216c46c3a17a0a3c252f11f2"
format_stamp: "Formatted at 2026-09-26 09:24:35 on dist-test-slave-7zlw"
server_key: "4ecd3763e18fcea9abe55d069aaa7bb1"
server_key_iv: "95b561643534c84c675c0330cefbbafb"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:35.371374 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "b0ca701a216c46c3a17a0a3c252f11f2"
format_stamp: "Formatted at 2026-09-26 09:24:35 on dist-test-slave-7zlw"
server_key: "4ecd3763e18fcea9abe55d069aaa7bb1"
server_key_iv: "95b561643534c84c675c0330cefbbafb"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:35.373190 5214 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 09:24:35.388325 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:35.388553 5214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:35.388672 5214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "b0ca701a216c46c3a17a0a3c252f11f2"
format_stamp: "Formatted at 2026-09-26 09:24:35 on dist-test-slave-7zlw"
server_key: "4ecd3763e18fcea9abe55d069aaa7bb1"
server_key_iv: "95b561643534c84c675c0330cefbbafb"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:35.388789 5214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:35.415952 5214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:35.418186 5214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:35.418372 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:35.423502 5214 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36261
I20260926 09:24:35.423576 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36261 every 8 connection(s)
I20260926 09:24:35.423957 5214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:35.424767 5282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:35.427757 5282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Bootstrap starting.
I20260926 09:24:35.428464 5282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:35.428778 5282 log.cc:824] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:35.429617 5282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: No bootstrap required, opened a new log
I20260926 09:24:35.430204 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5214
I20260926 09:24:35.430317 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:24:35.430485 25517 external_mini_cluster.cc:1428] Setting key 64e71d49cba5e48381cf772cb080519b
I20260926 09:24:35.432416 5282 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:35.432667 5282 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:35.432787 5282 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ca701a216c46c3a17a0a3c252f11f2, State: Initialized, Role: FOLLOWER
I20260926 09:24:35.432945 5282 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:35.433043 5282 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:35.433103 5282 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:35.433312 5282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:35.434137 5282 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:35.434249 5282 leader_election.cc:302] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ca701a216c46c3a17a0a3c252f11f2; no voters:
I20260926 09:24:35.434468 5282 leader_election.cc:288] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:24:35.434516 5287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:35.434826 5282 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:35.435541 5287 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 LEADER]: Becoming Leader. State: Replica: b0ca701a216c46c3a17a0a3c252f11f2, State: Running, Role: LEADER
I20260926 09:24:35.435645 5287 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:35.435956 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ca701a216c46c3a17a0a3c252f11f2. Latest consensus state: current_term: 1 leader_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } } }
I20260926 09:24:35.436017 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:35.436918 5282 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 09:24:35 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:35.439217 5288 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } } }
I20260926 09:24:35.439543 5288 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:35.439826 5295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:35.440966 5295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:35.444830 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:35.431051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.24.235.126:36261)
0926 09:24:35.431233 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:35.431243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:35.431275 (+ 32us) server_negotiation.cc:407] Connection header received
0926 09:24:35.432410 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:35.432419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:35.432473 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:35.432641 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:35.433497 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:35.434584 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:35.435410 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:35.436234 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:35.436347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:35.439621 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:35.439687 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:35.439687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:35.439849 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:35.442836 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:35.443478 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:35.443482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:35.443483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:35.443593 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:35.443812 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:35.443813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:35.443813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:35.444167 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:35.444328 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:35.444425 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 09:24:35.444584 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260926 09:24:35.449153 5295 catalog_manager.cc:1347] Generated new cluster ID: d484c7fc7f8941cd93043e0c03edd823
I20260926 09:24:35.449225 5295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:35.476106 5295 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:24:35.476930 5295 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:35.482153 5295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Generated new TSK 0
I20260926 09:24:35.482434 5295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:24:36.176749 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:36.276381 5327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:36.276684 5327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:36.276723 5327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:36.278497 5327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:36.278575 5327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:36.278618 5327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:24:36.280843 5327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:36.281134 5327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:36.281446 5327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:36.288024 5335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.288134 5327 server_base.cc:1034] running on GCE node
W20260926 09:24:36.288390 5333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.289249 5332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.289561 5327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:36.289861 5327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:36.291072 5327 hybrid_clock.cc:648] HybridClock initialized: now 1790414676291054 us; error 40 us; skew 500 ppm
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:36.294687 5327 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:24:36.295233 5327 webserver.cc:460] Webserver started at http://127.24.235.65:33905/ using document root <none> and password file <none>
I20260926 09:24:36.295501 5327 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:36.295809 5327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:36.296001 5327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:36.314910 5327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "8225d808913346cabd00d60ce47c959a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "c3c042e2d649ea25bf78e12c1063c2b2"
server_key_iv: "ac9163f352128fd3162f31322e00019d"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.315454 5327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "8225d808913346cabd00d60ce47c959a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "c3c042e2d649ea25bf78e12c1063c2b2"
server_key_iv: "ac9163f352128fd3162f31322e00019d"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.318531 5327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 09:24:36.336151 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.336468 5327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 09:24:36.336601 5327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "8225d808913346cabd00d60ce47c959a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "c3c042e2d649ea25bf78e12c1063c2b2"
server_key_iv: "ac9163f352128fd3162f31322e00019d"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.336737 5327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:36.350370 5327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:36.352929 5327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:36.353144 5327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:36.353423 5327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:36.353878 5327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:36.353941 5327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.353986 5327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:36.354014 5327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.361508 5327 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:38319
I20260926 09:24:36.361750 5456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:38319 every 8 connection(s)
I20260926 09:24:36.362211 5327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:24:36.365370 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5327
I20260926 09:24:36.365463 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:24:36.365682 25517 external_mini_cluster.cc:1428] Setting key e9ea68c8fc63c00f9552cb063a49e898
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
I20260926 09:24:36.379957 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.363497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.65:42927)
0926 09:24:36.364506 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:36.364546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:36.365247 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:36.365624 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:36.365628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:36.365885 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:36.366151 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.366157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.368547 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.368548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.369984 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.369987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.373359 (+ 3372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.373361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.373562 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.374287 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:36.374299 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:36.376103 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:36.378342 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.378351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.378354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.378605 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.378848 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.378849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.378849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.379011 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.379364 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:36.379372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:36.379562 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 09:24:36.379654 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:24:36.379780 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":83,"threads_started":1}
I20260926 09:24:36.382015 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.363813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:42927 (local address 127.24.235.126:36261)
0926 09:24:36.364000 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:36.364002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:36.364659 (+ 657us) server_negotiation.cc:407] Connection header received
0926 09:24:36.365414 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:36.365415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:36.365431 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:36.365494 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:36.366252 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.368411 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.372773 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.373209 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.376243 (+ 3034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.376315 (+ 72us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:36.376316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:36.376317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:36.376323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:36.378186 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.378701 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.378702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.378703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.378768 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.379091 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.379092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.379092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.379284 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:36.381726 (+ 2442us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:36.381777 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 09:24:36.381884 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260926 09:24:36.382671 5457 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:36.382774 5457 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:36.382977 5457 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:36.383491 5246 ts_manager.cc:194] Registered new tserver with Master: 8225d808913346cabd00d60ce47c959a (127.24.235.65:38319)
I20260926 09:24:36.384478 5246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:42927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:24:36.399580 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:36.512102 5464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:36.512344 5464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:36.512384 5464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:36.513655 5464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:36.513720 5464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:36.513763 5464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:24:36.515422 5464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:36.515699 5464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:36.515954 5464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:36.521277 5472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.521488 5464 server_base.cc:1034] running on GCE node
W20260926 09:24:36.521277 5471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.521716 5474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.521992 5464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:36.522269 5464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:36.523420 5464 hybrid_clock.cc:648] HybridClock initialized: now 1790414676523402 us; error 35 us; skew 500 ppm
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:36.526075 5464 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:24:36.526557 5464 webserver.cc:460] Webserver started at http://127.24.235.66:38749/ using document root <none> and password file <none>
I20260926 09:24:36.526803 5464 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:36.526871 5464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:36.527009 5464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:36.539062 5464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "a6ef8050649b4585b76301709bed85b0"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "000e882cdc1da7bd296b5d0759b161f6"
server_key_iv: "15143203b0cddd30969eff5cb0c4b769"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.539577 5464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "a6ef8050649b4585b76301709bed85b0"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "000e882cdc1da7bd296b5d0759b161f6"
server_key_iv: "15143203b0cddd30969eff5cb0c4b769"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.550350 5464 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.003s sys 0.000s
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:36.562362 5481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.562559 5464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:36.562645 5464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "a6ef8050649b4585b76301709bed85b0"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "000e882cdc1da7bd296b5d0759b161f6"
server_key_iv: "15143203b0cddd30969eff5cb0c4b769"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.562731 5464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:36.589391 5464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:36.591835 5464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:36.592056 5464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:36.592420 5464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:36.592839 5464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:36.592945 5464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.593047 5464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:36.593113 5464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.600284 5464 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:34337
I20260926 09:24:36.600541 5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:34337 every 8 connection(s)
I20260926 09:24:36.600819 5464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:24:36.606084 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5464
I20260926 09:24:36.606209 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:24:36.606413 25517 external_mini_cluster.cc:1428] Setting key 2a24a206f6378d970341772d739b4bdc
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:36.612028 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.602450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.66:47733)
0926 09:24:36.602608 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:36.602658 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:36.603168 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:36.603426 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:36.603430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:36.603620 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:36.603821 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.603824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.604584 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.604585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.605456 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.605458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.605875 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.605876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.606041 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.606841 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:36.606851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:36.608626 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:36.610321 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.610328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.610330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.610632 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.610855 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.610855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.610855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.610985 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.611312 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:36.611318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:36.611481 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 09:24:36.611583 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 09:24:36.611744 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 09:24:36.612555 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.602445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:47733 (local address 127.24.235.126:36261)
0926 09:24:36.602497 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:36.602500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:36.602758 (+ 258us) server_negotiation.cc:407] Connection header received
0926 09:24:36.603273 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:36.603274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:36.603291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:36.603343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:36.603918 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.604449 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.605544 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.605764 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.605853 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.608754 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:36.608756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:36.608757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:36.608765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:36.610219 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.610723 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.610725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.610726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.610786 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.611066 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.611067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.611067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.611225 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:36.611286 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:36.612320 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0926 09:24:36.612417 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:24:36.612988 5595 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:36.613133 5595 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:36.613310 5595 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:36.613731 5246 ts_manager.cc:194] Registered new tserver with Master: a6ef8050649b4585b76301709bed85b0 (127.24.235.66:34337)
I20260926 09:24:36.614099 5246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:47733
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:24:36.637126 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:36.729146 5601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:36.729496 5601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:36.729592 5601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:36.731273 5601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:36.731348 5601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:36.731395 5601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:24:36.733238 5601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:36.733560 5601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:36.733811 5601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:36.740464 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.741372 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.741472 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.742823 5601 server_base.cc:1034] running on GCE node
I20260926 09:24:36.743040 5601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:36.743330 5601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:36.746608 5601 hybrid_clock.cc:648] HybridClock initialized: now 1790414676746573 us; error 44 us; skew 500 ppm
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:36.749303 5601 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:24:36.749771 5601 webserver.cc:460] Webserver started at http://127.24.235.67:41647/ using document root <none> and password file <none>
I20260926 09:24:36.750061 5601 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:36.750128 5601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:36.750285 5601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:36.763422 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "77eb1d732a414090b08937030e8eca3a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "72a1f12b5c97d2ac474d87bfb9cdf006"
server_key_iv: "d8c3ffd9b481d864d97b5204c7b915d6"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.763911 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "77eb1d732a414090b08937030e8eca3a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "72a1f12b5c97d2ac474d87bfb9cdf006"
server_key_iv: "d8c3ffd9b481d864d97b5204c7b915d6"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.770956 5601 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260926 09:24:36.792577 5616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.792833 5601 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 09:24:36.792927 5601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "77eb1d732a414090b08937030e8eca3a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "72a1f12b5c97d2ac474d87bfb9cdf006"
server_key_iv: "d8c3ffd9b481d864d97b5204c7b915d6"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:36.793090 5601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:36.815649 5601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:36.818048 5601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:36.818222 5601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:36.818570 5601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:36.819007 5601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:36.819068 5601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.819098 5601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:36.819118 5601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:36.841074 5601 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:45611
I20260926 09:24:36.841550 5601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:24:36.843704 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5601
I20260926 09:24:36.843832 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:24:36.844081 25517 external_mini_cluster.cc:1428] Setting key 588bdb0176bdf8866d67ad9593e7da2c
I20260926 09:24:36.845314 5729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:45611 every 8 connection(s)
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:36.858109 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.846065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:44009 (local address 127.24.235.126:36261)
0926 09:24:36.846111 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:36.846113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:36.846623 (+ 510us) server_negotiation.cc:407] Connection header received
0926 09:24:36.847582 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:36.847584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:36.847596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:36.847646 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:36.848679 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.849356 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.850891 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.851118 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.851154 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.854555 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:36.854556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:36.854557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:36.854567 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:36.856410 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.856838 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.856840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.856840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.856907 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:36.857300 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:36.857301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:36.857301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:36.857447 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:36.857494 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:36.857921 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 09:24:36.858014 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:24:36.858412 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:36.846121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.67:44009)
0926 09:24:36.846449 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:36.846498 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:36.847459 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:36.847795 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:36.847801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:36.848179 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:36.848511 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.848517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.849457 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.849458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.850773 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:36.850776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:36.851298 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:36.851299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:36.851528 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:36.852499 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:36.852529 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:36.854422 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:36.856497 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.856505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.856507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.856754 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.856990 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:36.856991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:36.856991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:36.857203 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:36.857549 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:36.857556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:36.857729 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 09:24:36.857882 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 09:24:36.858056 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":190,"threads_started":1}
I20260926 09:24:36.859071 5730 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:36.859174 5730 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:36.859372 5730 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:36.859776 5246 ts_manager.cc:194] Registered new tserver with Master: 77eb1d732a414090b08937030e8eca3a (127.24.235.67:45611)
I20260926 09:24:36.860200 5246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:44009
I20260926 09:24:36.867466 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:24:36.868315 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5327
I20260926 09:24:36.874368 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5464
I20260926 09:24:36.879560 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5601
I20260926 09:24:36.884264 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5214
I20260926 09:24:36.889686 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36261
--webserver_interface=127.24.235.126
--webserver_port=37039
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36261
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:36.971757 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:36.971958 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:36.971980 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:36.973078 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:36.973112 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:36.973126 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:36.973138 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:36.973150 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:36.974723 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36261
--ranger_config_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:36.974905 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:36.975061 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:24:36.980726 5733 server_base.cc:1034] running on GCE node
W20260926 09:24:36.980646 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.980643 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:36.980640 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:36.981159 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:36.981463 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:36.982637 5733 hybrid_clock.cc:648] HybridClock initialized: now 1790414676982617 us; error 41 us; skew 500 ppm
Sep 26 09:24:36 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:36.985183 5733 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:36.985599 5733 webserver.cc:460] Webserver started at http://127.24.235.126:37039/ using document root <none> and password file <none>
I20260926 09:24:36.985831 5733 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:36.985905 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:37.047221 5733 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:37.048122 5749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.048379 5733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:37.048482 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "b0ca701a216c46c3a17a0a3c252f11f2"
format_stamp: "Formatted at 2026-09-26 09:24:35 on dist-test-slave-7zlw"
server_key: "4ecd3763e18fcea9abe55d069aaa7bb1"
server_key_iv: "95b561643534c84c675c0330cefbbafb"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:37.048832 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:37.064900 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:37.066975 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:37.067090 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:37.071465 5733 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36261
I20260926 09:24:37.071516 5801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36261 every 8 connection(s)
I20260926 09:24:37.071918 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:37.073348 5802 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:24:37.074009 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Bootstrap starting.
I20260926 09:24:37.075412 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5733
I20260926 09:24:37.075686 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:38319
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:36261
--webserver_port=33905
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:37.082617 5802 log.cc:824] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:37.084383 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:24:37.084702 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Bootstrap complete.
I20260926 09:24:37.087172 5802 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:37.087414 5802 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ca701a216c46c3a17a0a3c252f11f2, State: Initialized, Role: FOLLOWER
I20260926 09:24:37.087590 5802 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:37.087661 5802 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:37.087709 5802 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:37.087756 5802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:24:37.089213 5802 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:37.089330 5802 leader_election.cc:302] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ca701a216c46c3a17a0a3c252f11f2; no voters:
I20260926 09:24:37.089515 5802 leader_election.cc:288] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:24:37.089550 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:24:37.089658 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [term 2 LEADER]: Becoming Leader. State: Replica: b0ca701a216c46c3a17a0a3c252f11f2, State: Running, Role: LEADER
I20260926 09:24:37.089748 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } }
I20260926 09:24:37.089793 5802 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:37.090010 5806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } } }
I20260926 09:24:37.090085 5806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:37.090202 5806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ca701a216c46c3a17a0a3c252f11f2. Latest consensus state: current_term: 2 leader_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ca701a216c46c3a17a0a3c252f11f2" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36261 } } }
I20260926 09:24:37.090343 5806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:37.090729 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:37.091352 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:37.091516 5810 catalog_manager.cc:1259] Loaded cluster ID: d484c7fc7f8941cd93043e0c03edd823
I20260926 09:24:37.091607 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:37.091713 5802 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 09:24:37.092798 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:37.092972 5810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Loaded TSK: 0
I20260926 09:24:37.093227 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:24:37.186785 5804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:37.187074 5804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:37.187115 5804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:37.188939 5804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:37.188999 5804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:37.189065 5804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:24:37.191522 5804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:38319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=33905
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:37.191787 5804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:37.192103 5804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:37.198596 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.198709 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.198786 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:37.199838 5804 server_base.cc:1034] running on GCE node
I20260926 09:24:37.200078 5804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:37.200332 5804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:37.201488 5804 hybrid_clock.cc:648] HybridClock initialized: now 1790414677201453 us; error 48 us; skew 500 ppm
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:37.204737 5804 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:24:37.205310 5804 webserver.cc:460] Webserver started at http://127.24.235.65:33905/ using document root <none> and password file <none>
I20260926 09:24:37.205612 5804 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:37.205731 5804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:37.228147 5804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 09:24:37.229393 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.229640 5804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:37.229743 5804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "8225d808913346cabd00d60ce47c959a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "c3c042e2d649ea25bf78e12c1063c2b2"
server_key_iv: "ac9163f352128fd3162f31322e00019d"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:37.230293 5804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:37.248026 5804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:37.250124 5804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:37.250308 5804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:37.250543 5804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:37.250921 5804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:37.251016 5804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.251061 5804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:37.251101 5804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.258671 5804 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:38319
I20260926 09:24:37.258772 5963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:38319 every 8 connection(s)
I20260926 09:24:37.259224 5804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:24:37.263720 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5804
I20260926 09:24:37.264036 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:34337
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:36261
--webserver_port=38749
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:37.273262 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.260258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:48675 (local address 127.24.235.126:36261)
0926 09:24:37.260499 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:37.260510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:37.260711 (+ 201us) server_negotiation.cc:407] Connection header received
0926 09:24:37.261513 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:37.261520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:37.261586 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:37.261738 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:37.262471 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.263862 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.265469 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.266081 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.266238 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.268841 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:37.268845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:37.268845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:37.268858 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:37.271095 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.271634 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.271638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.271638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.271743 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.272017 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.272018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.272018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.272380 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:37.272473 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:37.272889 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 09:24:37.273044 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":119,"threads_started":1}
I20260926 09:24:37.273329 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.260164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.65:48675)
0926 09:24:37.260520 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:37.260567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:37.261259 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:37.261800 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:37.261805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:37.262043 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:37.262332 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.262338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.263991 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.263992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.265335 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.265338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.266230 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.266232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.266359 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.266994 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:37.267006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:37.268711 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:37.271209 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.271216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.271218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.271516 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.271826 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.271827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.271827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.271954 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.272506 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:37.272513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:37.272677 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 09:24:37.272808 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 09:24:37.272989 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":90,"threads_started":1}
I20260926 09:24:37.274466 5964 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:37.274564 5964 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.274760 5964 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.275727 5964 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:24:37.275794 5964 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.275849 5964 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:37.276221 5964 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.276291 5964 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.276676 5964 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:24:37.366422 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:37.366715 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:37.366758 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:37.368005 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:37.368084 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:37.368131 5968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:24:37.369840 5968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:34337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=38749
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:37.370122 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:37.370407 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:37.376407 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.377549 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.379694 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:37.380926 5968 server_base.cc:1034] running on GCE node
I20260926 09:24:37.381201 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:37.381505 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:37.382660 5968 hybrid_clock.cc:648] HybridClock initialized: now 1790414677382650 us; error 28 us; skew 500 ppm
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:37.385278 5968 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:24:37.385824 5968 webserver.cc:460] Webserver started at http://127.24.235.66:38749/ using document root <none> and password file <none>
I20260926 09:24:37.386080 5968 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:37.386170 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:37.403445 5968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:37.404316 5984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.404549 5968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:37.404721 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "a6ef8050649b4585b76301709bed85b0"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "000e882cdc1da7bd296b5d0759b161f6"
server_key_iv: "15143203b0cddd30969eff5cb0c4b769"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:37.405241 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:37.458943 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:37.461136 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:37.461318 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:37.461621 5968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:37.462088 5968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:37.462143 5968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.462176 5968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:37.462198 5968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.469836 5968 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:34337
I20260926 09:24:37.469919 6097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:34337 every 8 connection(s)
I20260926 09:24:37.470326 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:24:37.476212 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 5968
I20260926 09:24:37.476503 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.235.84:49507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:45611
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:36261
--webserver_port=41647
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:37.481319 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.471090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:60407 (local address 127.24.235.126:36261)
0926 09:24:37.471204 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:37.471206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:37.471406 (+ 200us) server_negotiation.cc:407] Connection header received
0926 09:24:37.472027 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:37.472029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:37.472043 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:37.472102 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:37.472692 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.473321 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.474445 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.474656 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.474731 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.477261 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:37.477262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:37.477263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:37.477270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:37.479010 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.480181 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.480182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.480183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.480244 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.480523 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.480524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.480525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.480647 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:37.480727 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:37.481104 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 09:24:37.481216 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 09:24:37.482152 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.471099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.66:60407)
0926 09:24:37.471277 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:37.471320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:37.471912 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:37.472173 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:37.472177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:37.472374 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:37.472572 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.472576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.473417 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.473418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.474355 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.474358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.474771 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.474772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.474903 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.475482 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:37.475498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:37.477149 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:37.479761 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.479768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.479770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.480078 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.480324 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.480325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.480325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.480446 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.480729 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:37.480736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:37.480881 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 09:24:37.481724 (+ 843us) client_negotiation.cc:240] Negotiation successful
0926 09:24:37.481905 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260926 09:24:37.482869 6098 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:37.483021 6098 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.483292 6098 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.483961 6098 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:24:37.484093 6098 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.484208 6098 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:37.484552 6098 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.484668 6098 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.485008 6098 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:24:37.576563 6101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:37.576874 6101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:37.576912 6101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:37.578152 6101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:37.578227 6101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:37.578272 6101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:24:37.579962 6101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.235.84:49507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:45611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=41647
--tserver_master_addrs=127.24.235.126:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:37.580230 6101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:37.580467 6101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:37.586892 6107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.586912 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:37.587162 6106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:37.590375 6101 server_base.cc:1034] running on GCE node
I20260926 09:24:37.590617 6101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:37.590915 6101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:37.592069 6101 hybrid_clock.cc:648] HybridClock initialized: now 1790414677592037 us; error 48 us; skew 500 ppm
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:37.594764 6101 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:24:37.595288 6101 webserver.cc:460] Webserver started at http://127.24.235.67:41647/ using document root <none> and password file <none>
I20260926 09:24:37.595554 6101 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:37.595644 6101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:37.614908 6101 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260926 09:24:37.616283 6118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.617712 6101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 09:24:37.617818 6101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "77eb1d732a414090b08937030e8eca3a"
format_stamp: "Formatted at 2026-09-26 09:24:36 on dist-test-slave-7zlw"
server_key: "72a1f12b5c97d2ac474d87bfb9cdf006"
server_key_iv: "d8c3ffd9b481d864d97b5204c7b915d6"
server_key_version: "kuduclusterkey@0"
I20260926 09:24:37.618211 6101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:37.641932 6101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:37.644133 6101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:37.644312 6101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:37.644594 6101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:37.644981 6101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:37.645097 6101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.645176 6101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:37.645238 6101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:37.652770 6101 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:45611
I20260926 09:24:37.653337 6101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:24:37.653494 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6101
I20260926 09:24:37.654026 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:45611 every 8 connection(s)
Sep 26 09:24:37 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:37.663044 5967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.653901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.24.235.126:36261)
0926 09:24:37.653965 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:37.653968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:37.654645 (+ 677us) server_negotiation.cc:407] Connection header received
0926 09:24:37.654746 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:37.654748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:37.654759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:37.654819 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:37.655559 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.656254 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.657871 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.658139 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.658194 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.659950 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:37.659952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:37.659952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:37.659960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:37.661786 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.662170 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.662171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.662172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.662230 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.662459 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.662460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.662460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.662611 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:37.662773 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:37.662864 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 09:24:37.662944 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260926 09:24:37.665155 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.654249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:49845 (local address 127.24.235.126:36261)
0926 09:24:37.654380 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:37.654382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:37.654594 (+ 212us) server_negotiation.cc:407] Connection header received
0926 09:24:37.655212 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:37.655213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:37.655225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:37.655287 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:37.656010 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.656759 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.657813 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.658122 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.658188 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.660527 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:37.660528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:37.660529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:37.660536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:37.662365 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.663995 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.663996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.663997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.664058 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:37.664368 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:37.664369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:37.664370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:37.664510 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:37.664590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:37.664933 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 09:24:37.665050 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":87,"threads_started":1}
I20260926 09:24:37.665315 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:37.654258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36261 (local address 127.24.235.67:49845)
0926 09:24:37.654431 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:37.654480 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:37.655093 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:37.655372 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:37.655381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:37.655616 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:37.655900 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.655905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.656856 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.656857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.657707 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:37.657710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:37.658281 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:37.658282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:37.658411 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:37.658894 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:37.658902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:37.660409 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:37.662828 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.663340 (+ 512us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.663343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.663692 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.664147 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:37.664148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:37.664149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:37.664283 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:37.664595 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:37.664602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:37.664762 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 09:24:37.664883 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 09:24:37.665064 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260926 09:24:37.665817 6232 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36261
I20260926 09:24:37.665923 6232 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.666119 6232 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.666867 6232 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 09:24:37.666947 6232 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.667017 6232 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:37.667404 6232 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:37.667476 6232 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
W20260926 09:24:37.667810 6232 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 09:24:38 dist-test-slave-7zlw krb5kdc[4770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790414678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:38.276865 5964 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:38.277060 5964 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:38.277395 5766 ts_manager.cc:194] Registered new tserver with Master: 8225d808913346cabd00d60ce47c959a (127.24.235.65:38319)
I20260926 09:24:38.278190 5766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:48675
I20260926 09:24:38.485256 6098 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:38.485389 6098 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:38.485743 5766 ts_manager.cc:194] Registered new tserver with Master: a6ef8050649b4585b76301709bed85b0 (127.24.235.66:34337)
I20260926 09:24:38.486354 5766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:60407
Sep 26 09:24:38 dist-test-slave-7zlw krb5kdc[4770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for HTTP/127.24.235.84@KRBTEST.COM
I20260926 09:24:38.667989 6232 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:38.668170 6232 heartbeater.cc:511] Master 127.24.235.126:36261 requested a full tablet report, sending...
I20260926 09:24:38.668571 5766 ts_manager.cc:194] Registered new tserver with Master: 77eb1d732a414090b08937030e8eca3a (127.24.235.67:45611)
I20260926 09:24:38.669008 5766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:49845
I20260926 09:24:38.670804 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:24:38.682430 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.672330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.24.235.126:36261)
0926 09:24:38.672628 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.672630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.672765 (+ 135us) server_negotiation.cc:407] Connection header received
0926 09:24:38.672834 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.672835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.672846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.672913 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:38.675208 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.675980 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.677070 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.677350 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.677419 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.678974 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:38.678976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:38.678976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:38.678984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:38.680844 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:38.681309 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:38.681311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:38.681311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:38.681390 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:38.681627 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:38.681628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:38.681628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:38.681787 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:38.681896 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.682236 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.682323 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":60,"threads_started":1}
I20260926 09:24:38.686574 5766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:24:38.688947 5766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:24:38.710197 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.697865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45611 (local address 127.0.0.1:49500)
0926 09:24:38.701257 (+ 3392us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.701310 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:38.701376 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:38.703461 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:38.703462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:38.703464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:38.703506 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.703508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.705078 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.705079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.706320 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.706323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.709862 (+ 3539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.709863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.709959 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.709978 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 09:24:38.710033 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:24:38.710093 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3368,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260926 09:24:38.711232 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.698120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38319 (local address 127.0.0.1:44592)
0926 09:24:38.701472 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.701482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:38.701523 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:38.702041 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:38.702045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:38.702047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:38.702137 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.702139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.706844 (+ 4705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.706845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.707891 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.707894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.711011 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.711012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.711074 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.711078 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:24:38.711107 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 09:24:38.711149 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3338,"thread_start_us":107,"threads_started":1}
I20260926 09:24:38.711764 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.699076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.24.235.67:45611)
0926 09:24:38.701630 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.701635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.701646 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:24:38.701697 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.701699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.701718 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.703583 (+ 1865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:38.703682 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.704627 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.706425 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.709754 (+ 3329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.711526 (+ 1772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.711607 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.711635 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.711690 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":67,"threads_started":1}
I20260926 09:24:38.712213 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.698452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.24.235.65:38319)
0926 09:24:38.701860 (+ 3408us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.701864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.701873 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:24:38.701906 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.701908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.701924 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.702222 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:38.702309 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.704961 (+ 2652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.708049 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.708880 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.711981 (+ 3101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.712064 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.712092 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.712138 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3367,"thread_start_us":68,"threads_started":1}
I20260926 09:24:38.713121 6166 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 (DEFAULT_TABLE table=test-table [id=7e4ef8bfb621442daabdbb9943cdb25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:38.713194 5898 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 (DEFAULT_TABLE table=test-table [id=7e4ef8bfb621442daabdbb9943cdb25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:38.713538 6166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca088a65149b5845a0c1dc92f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:38.713537 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca088a65149b5845a0c1dc92f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:38.716816 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.697865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:34337 (local address 127.0.0.1:49966)
0926 09:24:38.701482 (+ 3617us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.701495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:38.701834 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:38.702142 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:38.702143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:38.702144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:38.702198 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.702200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.713137 (+ 10937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.713138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.714150 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.714153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.716567 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.716569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.716637 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.716641 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:24:38.716692 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:24:38.716737 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3225,"thread_start_us":98,"threads_started":1}
I20260926 09:24:38.717134 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.698461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.24.235.66:34337)
0926 09:24:38.701191 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.701197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.701562 (+ 365us) server_negotiation.cc:407] Connection header received
0926 09:24:38.701942 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.701945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.701962 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.702019 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:38.702280 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.703396 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.715541 (+ 12145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.716356 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.716440 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.716494 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.716970 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.717055 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2688,"thread_start_us":54,"threads_started":1}
I20260926 09:24:38.717778 6032 tablet_service.cc:1467] Processing CreateTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 (DEFAULT_TABLE table=test-table [id=7e4ef8bfb621442daabdbb9943cdb25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:38.718124 6032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2ca088a65149b5845a0c1dc92f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:38.729249 6265 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Bootstrap starting.
I20260926 09:24:38.730257 6265 tablet_bootstrap.cc:654] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:38.730720 6265 log.cc:824] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:38.731169 6266 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Bootstrap starting.
I20260926 09:24:38.731963 6266 tablet_bootstrap.cc:654] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:38.732285 6266 log.cc:824] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:38.732954 6265 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: No bootstrap required, opened a new log
I20260926 09:24:38.733060 6265 ts_tablet_manager.cc:1397] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 09:24:38.735473 6265 raft_consensus.cc:348] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.735682 6265 raft_consensus.cc:374] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:38.735721 6265 raft_consensus.cc:729] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77eb1d732a414090b08937030e8eca3a, State: Initialized, Role: FOLLOWER
I20260926 09:24:38.735939 6267 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Bootstrap starting.
I20260926 09:24:38.736640 6267 tablet_bootstrap.cc:654] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:38.736968 6267 log.cc:824] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:38.737529 6266 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: No bootstrap required, opened a new log
I20260926 09:24:38.737648 6266 ts_tablet_manager.cc:1397] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 09:24:38.737797 6267 tablet_bootstrap.cc:492] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: No bootstrap required, opened a new log
I20260926 09:24:38.737865 6267 ts_tablet_manager.cc:1397] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:24:38.739835 6266 raft_consensus.cc:348] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.740022 6266 raft_consensus.cc:374] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:38.740058 6266 raft_consensus.cc:729] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8225d808913346cabd00d60ce47c959a, State: Initialized, Role: FOLLOWER
I20260926 09:24:38.740303 6265 consensus_queue.cc:260] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.740602 6265 ts_tablet_manager.cc:1428] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260926 09:24:38.740532 6267 raft_consensus.cc:348] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.740684 6267 raft_consensus.cc:374] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:38.740720 6267 raft_consensus.cc:729] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ef8050649b4585b76301709bed85b0, State: Initialized, Role: FOLLOWER
I20260926 09:24:38.740736 6232 heartbeater.cc:503] Master 127.24.235.126:36261 was elected leader, sending a full tablet report...
I20260926 09:24:38.740761 6266 consensus_queue.cc:260] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.741026 6266 ts_tablet_manager.cc:1428] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:24:38.741186 5964 heartbeater.cc:503] Master 127.24.235.126:36261 was elected leader, sending a full tablet report...
I20260926 09:24:38.741786 6267 consensus_queue.cc:260] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.742139 6267 ts_tablet_manager.cc:1428] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 09:24:38.742287 6098 heartbeater.cc:503] Master 127.24.235.126:36261 was elected leader, sending a full tablet report...
W20260926 09:24:38.760901 5965 tablet.cc:2369] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:24:38.842914 6271 raft_consensus.cc:482] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:38.843102 6271 raft_consensus.cc:504] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.843495 6271 leader_election.cc:288] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6ef8050649b4585b76301709bed85b0 (127.24.235.66:34337), 8225d808913346cabd00d60ce47c959a (127.24.235.65:38319)
I20260926 09:24:38.847797 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.844113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:42733 (local address 127.24.235.65:38319)
0926 09:24:38.844151 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.844153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.844161 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:24:38.844190 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.844191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.844200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.844257 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:38.844559 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.845294 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.846794 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.847382 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.847434 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.847443 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.847672 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.847718 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:24:38.847823 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.844002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:37529 (local address 127.24.235.66:34337)
0926 09:24:38.844060 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.844062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.844246 (+ 184us) server_negotiation.cc:407] Connection header received
0926 09:24:38.844345 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.844347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.844356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.844387 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:38.844628 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.845349 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.846808 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.847361 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.847414 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.847424 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.847701 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.847748 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:24:38.848093 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.843705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:34337 (local address 127.24.235.67:37529)
0926 09:24:38.844191 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.844200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:38.844271 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:38.844454 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:38.844454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:38.844457 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:38.844542 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.844545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.845457 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.845458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.846693 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.846696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.847525 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.847526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.847604 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.847609 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:24:38.847947 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 09:24:38.847996 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":17,"thread_start_us":142,"threads_started":1}
I20260926 09:24:38.848554 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.843705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38319 (local address 127.24.235.67:42733)
0926 09:24:38.844026 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.844039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:38.844104 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:38.844383 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:38.844384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:38.844386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:38.844473 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.844475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.845406 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.845406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.846685 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:38.846688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.847522 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.847523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:38.847589 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.847593 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:24:38.848396 (+ 803us) client_negotiation.cc:240] Negotiation successful
0926 09:24:38.848472 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":4,"thread_start_us":153,"threads_started":1}
I20260926 09:24:38.848593 6052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca088a65149b5845a0c1dc92f87b2" candidate_uuid: "77eb1d732a414090b08937030e8eca3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ef8050649b4585b76301709bed85b0" is_pre_election: true
I20260926 09:24:38.848770 6052 raft_consensus.cc:2454] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77eb1d732a414090b08937030e8eca3a in term 0.
I20260926 09:24:38.848903 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca088a65149b5845a0c1dc92f87b2" candidate_uuid: "77eb1d732a414090b08937030e8eca3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8225d808913346cabd00d60ce47c959a" is_pre_election: true
I20260926 09:24:38.849102 5918 raft_consensus.cc:2454] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77eb1d732a414090b08937030e8eca3a in term 0.
I20260926 09:24:38.849079 6122 leader_election.cc:302] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77eb1d732a414090b08937030e8eca3a, a6ef8050649b4585b76301709bed85b0; no voters:
I20260926 09:24:38.849220 6271 raft_consensus.cc:2781] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:24:38.849269 6271 raft_consensus.cc:482] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:24:38.849296 6271 raft_consensus.cc:3037] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:38.850257 6271 raft_consensus.cc:504] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.850405 6271 leader_election.cc:288] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [CANDIDATE]: Term 1 election: Requested vote from peers a6ef8050649b4585b76301709bed85b0 (127.24.235.66:34337), 8225d808913346cabd00d60ce47c959a (127.24.235.65:38319)
I20260926 09:24:38.850689 6052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca088a65149b5845a0c1dc92f87b2" candidate_uuid: "77eb1d732a414090b08937030e8eca3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ef8050649b4585b76301709bed85b0"
I20260926 09:24:38.850710 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2ca088a65149b5845a0c1dc92f87b2" candidate_uuid: "77eb1d732a414090b08937030e8eca3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8225d808913346cabd00d60ce47c959a"
I20260926 09:24:38.850773 6052 raft_consensus.cc:3037] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:38.850788 5918 raft_consensus.cc:3037] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:38.851722 6052 raft_consensus.cc:2454] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77eb1d732a414090b08937030e8eca3a in term 1.
I20260926 09:24:38.851732 5918 raft_consensus.cc:2454] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77eb1d732a414090b08937030e8eca3a in term 1.
I20260926 09:24:38.851964 6119 leader_election.cc:302] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77eb1d732a414090b08937030e8eca3a, 8225d808913346cabd00d60ce47c959a; no voters:
I20260926 09:24:38.852062 6271 raft_consensus.cc:2781] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:38.852226 6271 raft_consensus.cc:686] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 1 LEADER]: Becoming Leader. State: Replica: 77eb1d732a414090b08937030e8eca3a, State: Running, Role: LEADER
I20260926 09:24:38.852329 6271 consensus_queue.cc:237] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } }
I20260926 09:24:38.853308 5766 catalog_manager.cc:5495] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77eb1d732a414090b08937030e8eca3a (127.24.235.67). New cstate: current_term: 1 leader_uuid: "77eb1d732a414090b08937030e8eca3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77eb1d732a414090b08937030e8eca3a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 } health_report { overall_health: UNKNOWN } } }
I20260926 09:24:38.881474 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.878156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.24.235.67:45611)
0926 09:24:38.878221 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:38.878223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:38.878232 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:24:38.878274 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:38.878275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:38.878297 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:38.878379 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:24:38.878546 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.879286 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.880467 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:38.880794 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:38.880865 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:38.881083 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:24:38.881225 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:24:38.881309 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:38.881371 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 09:24:38.881393 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:24:38.883683 6052 raft_consensus.cc:1261] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 1 FOLLOWER]: Refusing update from remote peer 77eb1d732a414090b08937030e8eca3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:38.883683 5918 raft_consensus.cc:1261] T bb2ca088a65149b5845a0c1dc92f87b2 P 8225d808913346cabd00d60ce47c959a [term 1 FOLLOWER]: Refusing update from remote peer 77eb1d732a414090b08937030e8eca3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:38.884107 6271 consensus_queue.cc:1035] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ef8050649b4585b76301709bed85b0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:38.884313 6276 consensus_queue.cc:1035] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8225d808913346cabd00d60ce47c959a" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:38.891151 6280 mvcc.cc:205] Tried to move back new op lower bound from 7333538524704624640 to 7333538524579860480. Current Snapshot: MvccSnapshot[applied={T|T < 7333538524704624640}]
I20260926 09:24:38.890319 6279 mvcc.cc:205] Tried to move back new op lower bound from 7333538524704624640 to 7333538524579860480. Current Snapshot: MvccSnapshot[applied={T|T < 7333538524704624640}]
I20260926 09:24:38.892843 6281 mvcc.cc:205] Tried to move back new op lower bound from 7333538524704624640 to 7333538524579860480. Current Snapshot: MvccSnapshot[applied={T|T < 7333538524704624640 or (T in {7333538524704624640})}]
I20260926 09:24:38.894707 5766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:38.894809 5766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:38.895629 5766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0ca701a216c46c3a17a0a3c252f11f2: Sending DeleteTablet for 3 replicas of tablet bb2ca088a65149b5845a0c1dc92f87b2
I20260926 09:24:38.896166 6032 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:38 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:49966
I20260926 09:24:38.896533 6166 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:38 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:49500
I20260926 09:24:38.896868 5898 tablet_service.cc:1514] Processing DeleteTablet for tablet bb2ca088a65149b5845a0c1dc92f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:38 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:44592
I20260926 09:24:38.897423 6288 tablet_replica.cc:330] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: stopping tablet replica
I20260926 09:24:38.897542 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5804
I20260926 09:24:38.897568 6289 tablet_replica.cc:330] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: stopping tablet replica
I20260926 09:24:38.901553 6288 raft_consensus.cc:2229] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:24:38.901750 6289 raft_consensus.cc:2229] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:24:38.901947 6289 raft_consensus.cc:2258] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:38.902117 6288 raft_consensus.cc:2258] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:38.902698 6289 ts_tablet_manager.cc:1905] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:38.903198 6288 ts_tablet_manager.cc:1905] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:38.907321 6288 ts_tablet_manager.cc:1918] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:38.908665 6289 ts_tablet_manager.cc:1918] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:38.908823 6289 log.cc:1196] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/bb2ca088a65149b5845a0c1dc92f87b2
I20260926 09:24:38.909039 6288 log.cc:1196] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/bb2ca088a65149b5845a0c1dc92f87b2
I20260926 09:24:38.909365 6289 ts_tablet_manager.cc:1939] T bb2ca088a65149b5845a0c1dc92f87b2 P 77eb1d732a414090b08937030e8eca3a: Deleting consensus metadata
I20260926 09:24:38.909436 6288 ts_tablet_manager.cc:1939] T bb2ca088a65149b5845a0c1dc92f87b2 P a6ef8050649b4585b76301709bed85b0: Deleting consensus metadata
I20260926 09:24:38.910023 5753 catalog_manager.cc:4841] TS a6ef8050649b4585b76301709bed85b0 (127.24.235.66:34337): tablet bb2ca088a65149b5845a0c1dc92f87b2 (table test-table [id=7e4ef8bfb621442daabdbb9943cdb25d]) successfully deleted
I20260926 09:24:38.910164 5752 catalog_manager.cc:4841] TS 77eb1d732a414090b08937030e8eca3a (127.24.235.67:45611): tablet bb2ca088a65149b5845a0c1dc92f87b2 (table test-table [id=7e4ef8bfb621442daabdbb9943cdb25d]) successfully deleted
W20260926 09:24:38.910508 5750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:38319 (error 108)
I20260926 09:24:38.911149 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5968
I20260926 09:24:38.917256 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6101
I20260926 09:24:38.919999 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:38.919820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38319 (local address 127.0.0.1:44594)
0926 09:24:38.919892 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:38.919955 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:24:38.920253 5750 catalog_manager.cc:4584] TS 8225d808913346cabd00d60ce47c959a (127.24.235.65:38319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb2ca088a65149b5845a0c1dc92f87b2: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38319: connect: Connection refused (error 111)
I20260926 09:24:38.924422 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 5733
I20260926 09:24:38.929986 25517 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 09:24:39.274963 25517 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 09:24:39.275053 25517 mini_ranger.cc:66] Stopping Ranger...
I20260926 09:24:39.619911 25517 mini_ranger.cc:68] Stopped Ranger
2026-09-26 09:24:39.620 UTC [4800] LOG: received smart shutdown request
2026-09-26 09:24:39.624 UTC [4800] LOG: background worker "logical replication launcher" (PID 4809) exited with exit code 1
2026-09-26 09:24:39.624 UTC [4804] LOG: shutting down
2026-09-26 09:24:39.844 UTC [4800] LOG: database system is shut down
2026-09-26T09:24:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51708 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:24:40 dist-test-slave-7zlw krb5kdc[6300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:24:40 dist-test-slave-7zlw krb5kdc[6300](info): set up 2 sockets
Sep 26 09:24:40 dist-test-slave-7zlw krb5kdc[6300](info): commencing operation
krb5kdc: starting...
W20260926 09:24:41.037456 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:24:41.168399 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:42427
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:41.252545 6313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:41.252774 6313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:41.252797 6313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:41.254009 6313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:41.254056 6313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:41.254071 6313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:41.254083 6313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:41.254096 6313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:41.255837 6313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:41.256107 6313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:41.256363 6313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:41.261492 6321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.261492 6318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.261601 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:41.261899 6313 server_base.cc:1034] running on GCE node
I20260926 09:24:41.262144 6313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:41.262395 6313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:41.263617 6313 hybrid_clock.cc:648] HybridClock initialized: now 1790414681263601 us; error 28 us; skew 500 ppm
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:41.266712 6313 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:41.267318 6313 webserver.cc:460] Webserver started at http://127.24.235.126:44977/ using document root <none> and password file <none>
I20260926 09:24:41.267566 6313 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:41.267640 6313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:41.267791 6313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:41.269001 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "cbfa511748574e0fa061cd6be0d89dfd"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "315b49a6b0c34a20f7298ac0fcecb46c"
server_key_iv: "2d7d0b893fe9528fffd57ab3d9a9d17a"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.269472 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "cbfa511748574e0fa061cd6be0d89dfd"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "315b49a6b0c34a20f7298ac0fcecb46c"
server_key_iv: "2d7d0b893fe9528fffd57ab3d9a9d17a"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.270853 6313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:41.271739 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:41.271976 6313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:41.272104 6313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "cbfa511748574e0fa061cd6be0d89dfd"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "315b49a6b0c34a20f7298ac0fcecb46c"
server_key_iv: "2d7d0b893fe9528fffd57ab3d9a9d17a"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.272183 6313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:41.284268 6313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:41.286469 6313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:41.286696 6313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:41.291146 6313 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:42427
I20260926 09:24:41.291256 6380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:42427 every 8 connection(s)
I20260926 09:24:41.291584 6313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:41.292554 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:41.294121 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6313
I20260926 09:24:41.294246 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:24:41.294462 25517 external_mini_cluster.cc:1428] Setting key 1b71638c9ae9600add03a0ead6c69e46
I20260926 09:24:41.294351 6381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.125@KRBTEST.COM" created.
I20260926 09:24:41.306432 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.295230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42427 (local address 127.0.0.1:46248)
0926 09:24:41.295448 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.295499 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.296220 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.296655 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.296657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.296936 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.297230 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.297236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.298299 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.298300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.299589 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.299592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.300111 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.300113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.300205 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.300782 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.300795 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.302633 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.304678 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.304685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.304688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.304886 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.305252 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.305253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.305253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.305340 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.305755 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.305760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.305834 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.305939 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.306188 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":89,"thread_start_us":89,"threads_started":1}
I20260926 09:24:41.306435 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.295417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.24.235.126:42427)
0926 09:24:41.295556 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.295561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.295670 (+ 109us) server_negotiation.cc:407] Connection header received
0926 09:24:41.296404 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.296406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.296434 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.296551 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.297346 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.298187 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.299693 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.300010 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.300115 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.302744 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.302750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.302751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.302758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.304509 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.305057 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.305060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.305061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.305159 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.305410 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.305410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.305411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.305679 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.305762 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.305975 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.306139 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260926 09:24:41.307224 6381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.307686 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.307496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38232)
0926 09:24:41.307565 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.307651 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 09:24:41.307840 6330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
I20260926 09:24:41.308305 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.308205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38244)
0926 09:24:41.308259 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.308286 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 09:24:41.308434 6381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:36289: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
I20260926 09:24:41.325414 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:36289
--webserver_interface=127.24.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:41.357689 6381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } attempt: 1
I20260926 09:24:41.358283 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.358134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38246)
0926 09:24:41.358198 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.358248 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:24:41.358759 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.358664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38250)
0926 09:24:41.358711 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.358739 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:24:41.358897 6381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:36289: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
W20260926 09:24:41.408953 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:41.409209 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:41.409231 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:41.410482 6388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:41.410526 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:41.410540 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:41.410554 6388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:41.410590 6388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:41.412083 6388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:41.412343 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:41.412583 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:41.417611 6394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.417755 6396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.417755 6393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:41.417939 6388 server_base.cc:1034] running on GCE node
I20260926 09:24:41.418169 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:41.418427 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:41.419600 6388 hybrid_clock.cc:648] HybridClock initialized: now 1790414681419593 us; error 35 us; skew 500 ppm
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:41.422327 6388 init.cc:377] Logged in from keytab as kudu/127.24.235.125@KRBTEST.COM (short username kudu)
I20260926 09:24:41.422941 6388 webserver.cc:460] Webserver started at http://127.24.235.125:44507/ using document root <none> and password file <none>
I20260926 09:24:41.423183 6388 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:41.423254 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:41.423408 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:41.424520 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data/instance:
uuid: "bca7379191f4468d924651064a572d50"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "829a3576943077a47b180bfecdd448ff"
server_key_iv: "65602af0650e583827172a80d36b454c"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.424894 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal/instance:
uuid: "bca7379191f4468d924651064a572d50"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "829a3576943077a47b180bfecdd448ff"
server_key_iv: "65602af0650e583827172a80d36b454c"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.426364 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:24:41.427119 6381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } attempt: 2
I20260926 09:24:41.427376 6403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:41.427575 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.427420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38264)
0926 09:24:41.427495 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.427544 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 09:24:41.427814 6388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:41.427917 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "bca7379191f4468d924651064a572d50"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "829a3576943077a47b180bfecdd448ff"
server_key_iv: "65602af0650e583827172a80d36b454c"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.428151 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.428048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38272)
0926 09:24:41.428104 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.428127 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:24:41.427996 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260926 09:24:41.428292 6381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:36289: Network error: Client connection negotiation failed: client connection to 127.24.235.125:36289: connect: Connection refused (error 111)
I20260926 09:24:41.438825 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:41.441340 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:41.441515 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:41.446234 6388 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.125:36289
I20260926 09:24:41.446275 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.125:36289 every 8 connection(s)
I20260926 09:24:41.446645 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
I20260926 09:24:41.447543 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:41.449318 6456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.450791 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6388
I20260926 09:24:41.450903 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-1/wal/instance
I20260926 09:24:41.451123 25517 external_mini_cluster.cc:1428] Setting key a8b01f5cbe1a5d8e513221d4e7fe62d5
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.124@KRBTEST.COM; defaulting to no policy
I20260926 09:24:41.460057 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.450142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.24.235.126:42427)
0926 09:24:41.450329 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.450331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.450345 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:24:41.451078 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.451079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.451090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.451151 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.452159 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.452726 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.453843 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.454054 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.454133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.456353 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.456355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.456356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.456366 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.458300 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.458802 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.458804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.458805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.458881 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.459169 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.459170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.459171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.459336 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.459426 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.459806 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.459905 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 09:24:41.460088 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.450030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42427 (local address 127.0.0.1:46250)
0926 09:24:41.450223 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.450272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.450911 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.451259 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.451264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.451699 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.452012 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.452018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.452865 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.452866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.453735 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.453738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.454140 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.454141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.454267 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.454826 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.454835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.456245 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.458402 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.458409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.458411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.458710 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.458971 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.458972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.458972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.459086 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.459436 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.459443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.459619 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.459759 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.459916 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260926 09:24:41.460865 6456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.124@KRBTEST.COM" created.
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:24:41.468982 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.461189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38284)
0926 09:24:41.461241 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.461252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.461294 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.461599 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.461600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.461730 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.461807 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.461808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.462749 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.462750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.463225 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.463226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.463708 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.463709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.463777 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.464227 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.464233 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.465680 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.467857 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.467857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.467858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.468023 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.468268 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.468269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.468269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.468323 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.468707 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.468708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.468751 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.468821 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.468901 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:24:41.469136 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.461269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.24.235.125:36289)
0926 09:24:41.461422 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.461426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.461441 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:24:41.461457 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.461458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.461469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.461549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.461866 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.462641 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.463304 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.463621 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.463709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.465773 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.465780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.465781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.465788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.467773 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.468111 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.468115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.468115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.468182 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.468393 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.468394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.468395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.468606 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.468701 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.468882 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.468976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260926 09:24:41.469903 6456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.470257 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.470138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39848)
0926 09:24:41.470188 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.470236 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:24:41.470424 6407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
I20260926 09:24:41.470875 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.470787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39860)
0926 09:24:41.470834 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.470859 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:24:41.471006 6456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:46573: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
I20260926 09:24:41.480806 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:46573
--webserver_interface=127.24.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:41.519446 6381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } attempt: 3
I20260926 09:24:41.520239 6456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } attempt: 1
I20260926 09:24:41.520635 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.520489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39876)
0926 09:24:41.520547 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.520590 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:24:41.521066 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.520940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39886)
0926 09:24:41.520989 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.521033 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 09:24:41.521147 6456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:46573: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:24:41.527642 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.519954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.24.235.125:36289)
0926 09:24:41.520024 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.520027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.520037 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:41.520101 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.520102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.520121 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.520189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.520668 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.521524 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.522239 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.522564 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.522637 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.524757 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.524758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.524759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.524765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.526162 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.526579 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.526581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.526581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.526667 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.526953 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.526954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.526954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.527125 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.527194 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.527468 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.527553 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:24:41.527753 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.519878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38300)
0926 09:24:41.519943 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.519955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.520025 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.520196 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.520198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.520399 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.520483 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.520490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.521647 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.521648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.522156 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.522158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.522641 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.522642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.522700 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.523185 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.523190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.524661 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.526244 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.526245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.526245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.526494 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.526802 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.526803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.526804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.526874 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.527225 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.527227 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.527313 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.527429 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.527602 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:24:41.528312 6381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.528723 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.528603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39896)
0926 09:24:41.528663 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.528695 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:24:41.529129 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.528989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39910)
0926 09:24:41.529083 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.529108 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 09:24:41.529230 6381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:46573: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
I20260926 09:24:41.560425 6381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } attempt: 1
I20260926 09:24:41.561005 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.560841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39918)
0926 09:24:41.560917 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.560965 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 09:24:41.561450 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.561352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39920)
0926 09:24:41.561399 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.561429 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:24:41.561618 6381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:46573: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
W20260926 09:24:41.566246 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:41.566455 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:41.566478 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:41.567646 6463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:41.567690 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:41.567704 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:41.567718 6463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:41.567729 6463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:41.569372 6463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:42427,127.24.235.125:36289,127.24.235.124:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:41.569653 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:41.569909 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:41.575256 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.575203 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:41.575210 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:41.575686 6463 server_base.cc:1034] running on GCE node
I20260926 09:24:41.575942 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:41.576299 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:41.577487 6463 hybrid_clock.cc:648] HybridClock initialized: now 1790414681577438 us; error 67 us; skew 500 ppm
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:41.580336 6463 init.cc:377] Logged in from keytab as kudu/127.24.235.124@KRBTEST.COM (short username kudu)
I20260926 09:24:41.580927 6463 webserver.cc:460] Webserver started at http://127.24.235.124:34091/ using document root <none> and password file <none>
I20260926 09:24:41.581202 6463 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:41.581276 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:41.581439 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:41.582535 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data/instance:
uuid: "ad9294a93cfc4f949e0b27cecf523c04"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "8a9f7a07872dae59e255db31c0e9ec3f"
server_key_iv: "c6f383205e69516d6458f8820349eba1"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.582921 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal/instance:
uuid: "ad9294a93cfc4f949e0b27cecf523c04"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "8a9f7a07872dae59e255db31c0e9ec3f"
server_key_iv: "c6f383205e69516d6458f8820349eba1"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.584383 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:24:41.585322 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:41.585592 6463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:41.585716 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "ad9294a93cfc4f949e0b27cecf523c04"
format_stamp: "Formatted at 2026-09-26 09:24:41 on dist-test-slave-7zlw"
server_key: "8a9f7a07872dae59e255db31c0e9ec3f"
server_key_iv: "c6f383205e69516d6458f8820349eba1"
server_key_version: "encryptionkey@0"
I20260926 09:24:41.585821 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:41.589285 6456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } attempt: 2
I20260926 09:24:41.589761 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.589618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39924)
0926 09:24:41.589678 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.589715 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:24:41.590090 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.589995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39926)
0926 09:24:41.590042 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.590067 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:24:41.590188 6456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:46573: Network error: Client connection negotiation failed: client connection to 127.24.235.124:46573: connect: Connection refused (error 111)
I20260926 09:24:41.605118 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:41.607707 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:41.607880 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:41.612298 6463 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.124:46573
I20260926 09:24:41.612355 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.124:46573 every 8 connection(s)
I20260926 09:24:41.612762 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
I20260926 09:24:41.613646 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:41.615274 6531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.616362 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6463
I20260926 09:24:41.616487 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790414522268193-25517-0/minicluster-data/master-2/wal/instance
I20260926 09:24:41.616699 25517 external_mini_cluster.cc:1428] Setting key a0b5502dad078473c87ff11beac3c615
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:24:41.626575 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.615972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.24.235.126:42427)
0926 09:24:41.616099 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.616102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.616298 (+ 196us) server_negotiation.cc:407] Connection header received
0926 09:24:41.616914 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.616915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.616925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.616998 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.617879 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.618506 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.619802 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.620131 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.620205 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.622981 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.622983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.622983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.622990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.624678 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.625172 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.625173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.625174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.625236 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.625751 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.625752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.625752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.625932 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.626016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.626383 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.626470 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 09:24:41.626705 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.615927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:42427 (local address 127.0.0.1:46258)
0926 09:24:41.616162 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.616210 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.616800 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.617103 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.617108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.617519 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.617773 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.617778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.618602 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.618602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.619722 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.619724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.620274 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.620276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.620430 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.621142 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.621154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.622882 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.624801 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.624807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.624809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.625102 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.625331 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.625331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.625332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.625618 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.626022 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.626030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.626216 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.626297 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.626459 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":186,"thread_start_us":125,"threads_started":1}
I20260926 09:24:41.626814 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.617098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.24.235.124:46573)
0926 09:24:41.617274 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.617279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.617341 (+ 62us) server_negotiation.cc:407] Connection header received
0926 09:24:41.617389 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.617392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.617415 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.617504 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.618242 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.619316 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.620288 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.620754 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.620843 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.623479 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.623486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.623486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.623492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.625345 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.625887 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.625890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.625890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.625950 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.626154 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.626155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.626155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.626324 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.626479 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.626575 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.626727 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260926 09:24:41.627419 6531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:41.635614 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.627767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.24.235.125:36289)
0926 09:24:41.627833 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.627836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.627846 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:41.627874 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.627875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.627887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.627938 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.628319 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.628997 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.629972 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.630204 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.630269 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.632821 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.632822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.632823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.632832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.634485 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.634837 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.634838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.634838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.634897 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.635107 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.635107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.635108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.635222 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.635293 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.635453 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.635524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 09:24:41.635619 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.627708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:36289 (local address 127.0.0.1:38308)
0926 09:24:41.627745 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.627756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.627796 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.627986 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.627987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.628140 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.628207 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.628209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.629147 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.629148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.629887 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.629889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.630291 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.630292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.630366 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.630736 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.630740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.632707 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.634580 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.634581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.634581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.634753 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.634965 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.634966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.634966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.635023 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.635299 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.635300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.635341 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.635427 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.635518 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 09:24:41.635998 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.627891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.24.235.126:42427)
0926 09:24:41.627923 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.627925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.627934 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:24:41.627952 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.627953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.627960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.628009 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.628601 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.629268 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.630154 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.630406 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.630453 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.632855 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.632857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.632857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.632864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.634746 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.635165 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.635166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.635166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.635231 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.635445 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.635446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.635446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.635567 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.635770 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.635808 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.635883 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:24:41.636395 6531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:24:41.636776 6381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } attempt: 2
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:24:41.645197 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.636656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39940)
0926 09:24:41.636694 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.636706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.636749 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.637006 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.637007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.637186 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.637219 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.637221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.638112 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.638113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.638739 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.638741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.639209 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.639265 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.639682 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.639688 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.641307 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.643993 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.643994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.643994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.644301 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.644474 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.644475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.644555 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.644883 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.644883 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.644918 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.644982 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.645091 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 09:24:41.645212 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.636880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.24.235.125:36289)
0926 09:24:41.637156 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.637157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.637163 (+ 6us) server_negotiation.cc:407] Connection header received
0926 09:24:41.637177 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.637178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.637184 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.637256 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.637814 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.638468 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639120 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.639401 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639461 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.642274 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.642275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.642276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.642281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.643784 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644186 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644249 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644477 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644619 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.644802 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.644991 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.645095 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260926 09:24:41.645359 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.636998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39952)
0926 09:24:41.637057 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.637066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.637104 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.637367 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.637368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.637506 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.637534 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.637536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.638425 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.638426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.639165 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.639168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.639618 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.639678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.640064 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.640069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.642131 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.644294 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.644296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.644503 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.644694 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.644695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.644755 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.645073 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.645074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.645117 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.645168 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.645261 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:24:41.645457 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.636688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.24.235.124:46573)
0926 09:24:41.636884 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.636885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.636893 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:24:41.636911 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.636912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.636928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.636976 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.637300 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.638035 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.638821 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.639153 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639214 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.641389 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.641390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.641390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.641396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.643909 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644358 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644407 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644619 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644783 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.644862 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.645306 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.645367 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260926 09:24:41.645771 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.637058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.24.235.124:46573)
0926 09:24:41.637262 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.637263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.637270 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:24:41.637288 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.637289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.637297 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.637352 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.637609 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.638341 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639249 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.639520 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.639612 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.642247 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.642249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.642249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.642258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.644217 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644576 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644628 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.644824 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.644825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.644825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.644942 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.645533 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.645573 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.645646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":131,"threads_started":1}
I20260926 09:24:41.646876 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:24:41.647445 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: Bootstrap starting.
I20260926 09:24:41.647698 6381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: Bootstrap starting.
I20260926 09:24:41.648285 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:41.648360 6381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:41.648643 6381 log.cc:824] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:41.648643 6531 log.cc:824] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:41.649554 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: No bootstrap required, opened a new log
I20260926 09:24:41.651685 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.652302 6381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: No bootstrap required, opened a new log
I20260926 09:24:41.652343 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:41.652395 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9294a93cfc4f949e0b27cecf523c04, State: Initialized, Role: FOLLOWER
I20260926 09:24:41.652519 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.653327 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:41.653577 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.653680 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.654660 6381 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.654807 6381 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:41.654845 6381 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfa511748574e0fa061cd6be0d89dfd, State: Initialized, Role: FOLLOWER
I20260926 09:24:41.654954 6381 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.655254 6381 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:41.656164 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.656312 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.656844 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.648412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.24.235.125:36289)
0926 09:24:41.648448 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.648450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.648618 (+ 168us) server_negotiation.cc:407] Connection header received
0926 09:24:41.648642 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.648643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.648653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.648700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.649498 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.650235 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.651234 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.651517 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.651574 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.653277 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.653279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.653279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.653287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.655049 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.655652 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.655653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.655654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.655718 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.655943 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.655944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.655944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.656394 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.656494 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.656658 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.656744 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:24:41.657328 6560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:24:41.657410 6560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:24:41.656869 6463 master_runner.cc:186] Error getting master registration for 127.24.235.126:42427: OK, instance_id { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" instance_seqno: 1790414681289877 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:24:41.660127 6463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:24:41.657912 6313 master_runner.cc:186] Error getting master registration for 127.24.235.125:36289: OK, instance_id { permanent_uuid: "bca7379191f4468d924651064a572d50" instance_seqno: 1790414681444785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:24:41.660359 6313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:24:41.660797 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.648607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.24.235.124:46573)
0926 09:24:41.648638 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.648640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.649208 (+ 568us) server_negotiation.cc:407] Connection header received
0926 09:24:41.649306 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.649307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.649320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.649349 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.650310 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.650996 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.651917 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.652260 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.652329 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.653738 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.653739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.653739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.653746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.655502 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.655975 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.655977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.655977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.656055 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.656417 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.656418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.656419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.660290 (+ 3871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.660397 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.660629 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.660716 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:24:41.661725 6569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:24:41.661816 6569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:24:41.662137 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.648311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.24.235.126:42427)
0926 09:24:41.648350 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.648352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.648870 (+ 518us) server_negotiation.cc:407] Connection header received
0926 09:24:41.648965 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.648966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.648977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.649006 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.649657 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.650322 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.651254 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.651552 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.651587 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.658944 (+ 7357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.658946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.658946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.658953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.660637 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.661114 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.661116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.661116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.661181 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.661595 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.661596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.661596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.661709 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.661807 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.661968 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.662043 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:24:41.681327 6456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } attempt: 3
Sep 26 09:24:41 dist-test-slave-7zlw krb5kdc[6300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:24:41.689862 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.681654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:46573 (local address 127.0.0.1:39968)
0926 09:24:41.681703 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:41.681715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:41.681753 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:41.681904 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:41.681905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:41.682097 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:41.682152 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.682154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.683092 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.683093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.683910 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:41.683912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.684390 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.684391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:41.684453 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.684992 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:41.684997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:41.686629 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:41.688607 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.688608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.688609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.688832 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.689144 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:41.689145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:41.689146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:41.689220 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:41.689517 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:41.689518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:41.689568 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:24:41.689652 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:24:41.689754 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 09:24:41.689880 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:41.681716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.24.235.124:46573)
0926 09:24:41.681761 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:41.681763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:41.681775 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:24:41.681816 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:41.681817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:41.681826 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:41.681881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:41.682233 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.683004 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.683997 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:41.684303 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:41.684375 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:41.686745 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:41.686747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:41.686747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:41.686754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:41.688512 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.688994 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.688995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.688995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.689075 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:41.689288 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:41.689289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:41.689289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:41.689425 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:41.689492 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:41.689692 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 09:24:41.689774 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:24:41.691500 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: Bootstrap starting.
I20260926 09:24:41.692277 6456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:41.692625 6456 log.cc:824] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:41.693571 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: No bootstrap required, opened a new log
I20260926 09:24:41.695649 6456 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.695796 6456 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:41.695829 6456 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca7379191f4468d924651064a572d50, State: Initialized, Role: FOLLOWER
I20260926 09:24:41.696019 6456 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.696393 6456 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 09:24:41.698432 6584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:24:41.698488 6584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:24:41.700719 6388 master_runner.cc:263] No leader master found from master bca7379191f4468d924651064a572d50
I20260926 09:24:41.700764 6388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:24:41.696362 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.700938 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.701049 6573 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:41.701165 6573 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.701637 6355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca7379191f4468d924651064a572d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfa511748574e0fa061cd6be0d89dfd" is_pre_election: true
I20260926 09:24:41.701772 6355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bca7379191f4468d924651064a572d50 in term 0.
I20260926 09:24:41.702044 6404 leader_election.cc:302] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bca7379191f4468d924651064a572d50, cbfa511748574e0fa061cd6be0d89dfd; no voters:
I20260926 09:24:41.702440 6505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca7379191f4468d924651064a572d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9294a93cfc4f949e0b27cecf523c04" is_pre_election: true
I20260926 09:24:41.702595 6505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bca7379191f4468d924651064a572d50 in term 0.
I20260926 09:24:41.702836 6573 leader_election.cc:288] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbfa511748574e0fa061cd6be0d89dfd (127.24.235.126:42427), ad9294a93cfc4f949e0b27cecf523c04 (127.24.235.124:46573)
I20260926 09:24:41.702950 6573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:24:41.703032 6573 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:24:41.703100 6573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:41.704056 6573 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.704350 6355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca7379191f4468d924651064a572d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfa511748574e0fa061cd6be0d89dfd"
I20260926 09:24:41.704443 6355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:41.705318 6355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca7379191f4468d924651064a572d50 in term 1.
I20260926 09:24:41.705541 6404 leader_election.cc:302] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bca7379191f4468d924651064a572d50, cbfa511748574e0fa061cd6be0d89dfd; no voters:
I20260926 09:24:41.705768 6505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca7379191f4468d924651064a572d50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9294a93cfc4f949e0b27cecf523c04"
I20260926 09:24:41.705840 6505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:41.706693 6505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca7379191f4468d924651064a572d50 in term 1.
I20260926 09:24:41.706947 6573 leader_election.cc:288] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [CANDIDATE]: Term 1 election: Requested vote from peers cbfa511748574e0fa061cd6be0d89dfd (127.24.235.126:42427), ad9294a93cfc4f949e0b27cecf523c04 (127.24.235.124:46573)
I20260926 09:24:41.707047 6573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:41.707122 6573 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [term 1 LEADER]: Becoming Leader. State: Replica: bca7379191f4468d924651064a572d50, State: Running, Role: LEADER
I20260926 09:24:41.707237 6573 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } }
I20260926 09:24:41.707734 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca7379191f4468d924651064a572d50. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.707857 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:41.708128 6588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:41.708397 6588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:41.710353 6505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [term 1 FOLLOWER]: Refusing update from remote peer bca7379191f4468d924651064a572d50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:41.710747 6586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:41.712096 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca7379191f4468d924651064a572d50. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.712200 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.712404 6355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [term 1 FOLLOWER]: Refusing update from remote peer bca7379191f4468d924651064a572d50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:41.712729 6586 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:41.713941 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca7379191f4468d924651064a572d50. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.714023 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.715270 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.715356 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:41.715801 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.715873 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.716154 6589 mvcc.cc:205] Tried to move back new op lower bound from 7333538536283484160 to 7333538536273801216. Current Snapshot: MvccSnapshot[applied={T|T < 7333538536283484160}]
I20260926 09:24:41.716353 6592 mvcc.cc:205] Tried to move back new op lower bound from 7333538536283484160 to 7333538536273801216. Current Snapshot: MvccSnapshot[applied={T|T < 7333538536283484160}]
I20260926 09:24:41.717125 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.717199 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:41.717569 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca7379191f4468d924651064a572d50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfa511748574e0fa061cd6be0d89dfd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 42427 } } peers { permanent_uuid: "bca7379191f4468d924651064a572d50" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 36289 } } peers { permanent_uuid: "ad9294a93cfc4f949e0b27cecf523c04" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 46573 } } }
I20260926 09:24:41.717645 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:24:41.718691 6588 catalog_manager.cc:1347] Generated new cluster ID: 3222667041714073affda5971a80924e
I20260926 09:24:41.718736 6588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:41.730840 6590 mvcc.cc:205] Tried to move back new op lower bound from 7333538536283484160 to 7333538536273801216. Current Snapshot: MvccSnapshot[applied={T|T < 7333538536283484160}]
I20260926 09:24:41.735903 6588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:24:41.736747 6588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:41.748005 6588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca7379191f4468d924651064a572d50: Generated new TSK 0
I20260926 09:24:41.748364 6588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:24:42.657883 6560 catalog_manager.cc:1259] Loaded cluster ID: 3222667041714073affda5971a80924e
I20260926 09:24:42.657965 6560 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: loading cluster ID for follower catalog manager: success
I20260926 09:24:42.658885 6560 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: acquiring CA information for follower catalog manager: success
I20260926 09:24:42.659243 6560 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad9294a93cfc4f949e0b27cecf523c04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:24:42.662261 6569 catalog_manager.cc:1259] Loaded cluster ID: 3222667041714073affda5971a80924e
I20260926 09:24:42.662344 6569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: loading cluster ID for follower catalog manager: success
I20260926 09:24:42.663136 6569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: acquiring CA information for follower catalog manager: success
I20260926 09:24:42.663440 6569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbfa511748574e0fa061cd6be0d89dfd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:24:42.803689 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6313
I20260926 09:24:42.809602 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6388
I20260926 09:24:42.814765 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6463
2026-09-26T09:24:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2854 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:24:42 dist-test-slave-7zlw krb5kdc[6604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:24:42 dist-test-slave-7zlw krb5kdc[6604](info): set up 2 sockets
Sep 26 09:24:42 dist-test-slave-7zlw krb5kdc[6604](info): commencing operation
krb5kdc: starting...
W20260926 09:24:43.866055 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:24:43 dist-test-slave-7zlw krb5kdc[6604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:24:43.991914 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33313
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:44.072621 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:44.072887 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:44.072925 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:44.074113 6617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:44.074195 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:44.074218 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:44.074240 6617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:44.074271 6617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:44.075743 6617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:44.076015 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:44.076249 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:44.081247 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:44.081310 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:44.081310 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:44.081321 6617 server_base.cc:1034] running on GCE node
I20260926 09:24:44.081779 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:44.082062 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:44.083230 6617 hybrid_clock.cc:648] HybridClock initialized: now 1790414684083218 us; error 28 us; skew 500 ppm
Sep 26 09:24:44 dist-test-slave-7zlw krb5kdc[6604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414684, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:44.086006 6617 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:44.086494 6617 webserver.cc:460] Webserver started at http://127.24.235.126:38951/ using document root <none> and password file <none>
I20260926 09:24:44.086768 6617 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:44.086845 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:44.087003 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:44.088178 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "3febb1d802c147f3a963ff421e1e9eb1"
format_stamp: "Formatted at 2026-09-26 09:24:44 on dist-test-slave-7zlw"
server_key: "adaf7e5e6606d7793743b9cfdd2fc3c0"
server_key_iv: "e0f78d46a400ef8f2a53dc8f6647f69d"
server_key_version: "encryptionkey@0"
I20260926 09:24:44.088599 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "3febb1d802c147f3a963ff421e1e9eb1"
format_stamp: "Formatted at 2026-09-26 09:24:44 on dist-test-slave-7zlw"
server_key: "adaf7e5e6606d7793743b9cfdd2fc3c0"
server_key_iv: "e0f78d46a400ef8f2a53dc8f6647f69d"
server_key_version: "encryptionkey@0"
I20260926 09:24:44.090188 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:24:44.091102 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:44.091357 6617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:24:44.091476 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "3febb1d802c147f3a963ff421e1e9eb1"
format_stamp: "Formatted at 2026-09-26 09:24:44 on dist-test-slave-7zlw"
server_key: "adaf7e5e6606d7793743b9cfdd2fc3c0"
server_key_iv: "e0f78d46a400ef8f2a53dc8f6647f69d"
server_key_version: "encryptionkey@0"
I20260926 09:24:44.091557 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:44.098980 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:44.101256 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:44.101433 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:44.105985 6617 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33313
I20260926 09:24:44.106015 6684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33313 every 8 connection(s)
I20260926 09:24:44.106403 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:44.107138 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6617
I20260926 09:24:44.107244 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:24:44.107456 25517 external_mini_cluster.cc:1428] Setting key 878554744c2cfd531d6993e5f705e9ea
I20260926 09:24:44.107468 6685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:44.110915 6685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1: Bootstrap starting.
I20260926 09:24:44.111608 6685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:44.111935 6685 log.cc:824] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:44.112922 6685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1: No bootstrap required, opened a new log
Sep 26 09:24:44 dist-test-slave-7zlw krb5kdc[6604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:44.115247 6685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } }
I20260926 09:24:44.115435 6685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:44.115479 6685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3febb1d802c147f3a963ff421e1e9eb1, State: Initialized, Role: FOLLOWER
I20260926 09:24:44.115651 6685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } }
I20260926 09:24:44.115734 6685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:44.115757 6685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:44.115809 6685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:44.116473 6685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } }
I20260926 09:24:44.116560 6685 leader_election.cc:302] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3febb1d802c147f3a963ff421e1e9eb1; no voters:
I20260926 09:24:44.116815 6690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:44.116783 6685 leader_election.cc:288] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:24:44.116984 6690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [term 1 LEADER]: Becoming Leader. State: Replica: 3febb1d802c147f3a963ff421e1e9eb1, State: Running, Role: LEADER
I20260926 09:24:44.117154 6690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } }
I20260926 09:24:44.117230 6685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:44.117533 6691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3febb1d802c147f3a963ff421e1e9eb1. Latest consensus state: current_term: 1 leader_uuid: "3febb1d802c147f3a963ff421e1e9eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } } }
I20260926 09:24:44.117796 6691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:44.117709 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3febb1d802c147f3a963ff421e1e9eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3febb1d802c147f3a963ff421e1e9eb1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33313 } } }
I20260926 09:24:44.118116 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:44.118301 6699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:44.118816 6699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:44.119506 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:44.107985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.24.235.126:33313)
0926 09:24:44.108213 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:44.108222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:44.108261 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:24:44.108776 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:44.108784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:44.108835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:44.108957 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:44.109798 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:44.110765 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:44.111828 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:44.112298 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:44.112410 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:44.114986 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:44.114990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:44.114991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:44.115000 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:44.117351 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:44.117829 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:44.117832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:44.117832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:44.117950 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:44.118210 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:44.118211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:44.118211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:44.118554 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:44.118768 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:44.118932 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:24:44.119290 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":124,"threads_started":1}
I20260926 09:24:44.120857 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:24:44.125449 6699 catalog_manager.cc:1347] Generated new cluster ID: 822fb22314644b5b93cd0ac8452c5ed6
I20260926 09:24:44.125515 6699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:44.188926 6699 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:24:44.189548 6699 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:44.193809 6699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3febb1d802c147f3a963ff421e1e9eb1: Generated new TSK 0
I20260926 09:24:44.194034 6699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:24:44.225955 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6617
2026-09-26T09:24:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1409 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:24:44 dist-test-slave-7zlw krb5kdc[6712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:24:44 dist-test-slave-7zlw krb5kdc[6712](info): set up 2 sockets
Sep 26 09:24:44 dist-test-slave-7zlw krb5kdc[6712](info): commencing operation
krb5kdc: starting...
W20260926 09:24:45.269428 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:24:45.400450 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:45.483075 6725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:45.483306 6725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:45.483330 6725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:45.484632 6725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:45.484679 6725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:45.484694 6725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:45.484707 6725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:45.484719 6725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:45.486402 6725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:45.486678 6725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:45.486920 6725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:45.492095 6731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:45.492125 6730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:45.492290 6725 server_base.cc:1034] running on GCE node
W20260926 09:24:45.492125 6733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:45.492669 6725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:45.493062 6725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:45.494272 6725 hybrid_clock.cc:648] HybridClock initialized: now 1790414685494246 us; error 36 us; skew 500 ppm
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:45.497231 6725 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:45.497673 6725 webserver.cc:460] Webserver started at http://127.24.235.126:35479/ using document root <none> and password file <none>
I20260926 09:24:45.497923 6725 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:45.497974 6725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:45.498128 6725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:45.499341 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "4f4314861bb847899b76b29f48154992"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "d6a84c46b8d7373284922719654fba4b"
server_key_iv: "8a3b3dd11c40d93867cc6f8527619825"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.499742 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "4f4314861bb847899b76b29f48154992"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "d6a84c46b8d7373284922719654fba4b"
server_key_iv: "8a3b3dd11c40d93867cc6f8527619825"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.501325 6725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:24:45.502208 6740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.502427 6725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:45.502545 6725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "4f4314861bb847899b76b29f48154992"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "d6a84c46b8d7373284922719654fba4b"
server_key_iv: "8a3b3dd11c40d93867cc6f8527619825"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.502676 6725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:45.522756 6725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:45.525259 6725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:45.525422 6725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:45.529762 6725 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37637
I20260926 09:24:45.529800 6792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37637 every 8 connection(s)
I20260926 09:24:45.530187 6725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:45.531155 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:45.533584 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap starting.
I20260926 09:24:45.534184 6793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:45.534472 6793 log.cc:824] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:45.535179 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: No bootstrap required, opened a new log
I20260926 09:24:45.536032 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6725
I20260926 09:24:45.536136 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:24:45.536351 25517 external_mini_cluster.cc:1428] Setting key fc82666c92fd1d18aeb80d334f659061
I20260926 09:24:45.537416 6793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:45.537580 6793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:45.537616 6793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4314861bb847899b76b29f48154992, State: Initialized, Role: FOLLOWER
I20260926 09:24:45.537801 6793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:45.537899 6793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:45.537931 6793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:45.537976 6793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:45.538969 6793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:45.539057 6793 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4314861bb847899b76b29f48154992; no voters:
I20260926 09:24:45.539245 6793 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:24:45.539295 6798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:45.539438 6798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 LEADER]: Becoming Leader. State: Replica: 4f4314861bb847899b76b29f48154992, State: Running, Role: LEADER
I20260926 09:24:45.539505 6793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:45.539590 6798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:45.541169 6799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:45.542346 6799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:45.541316 6800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4314861bb847899b76b29f48154992. Latest consensus state: current_term: 1 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:45.543455 6800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
W20260926 09:24:45.541396 6813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:24:45.543601 6813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:24:45.543699 6815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:45.543896 6815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:45.546270 6815 catalog_manager.cc:1347] Generated new cluster ID: 55fce923d21a45b69f4bdf846f8bfd2a
I20260926 09:24:45.546326 6815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:45.548702 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:45.536944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.24.235.126:37637)
0926 09:24:45.537139 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:45.537148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:45.537177 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:24:45.537396 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:45.537403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:45.537452 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:45.537598 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:45.538447 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.539774 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.540950 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.541604 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.541696 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:45.544817 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:45.544821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:45.544821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:45.544833 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:45.546962 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.547390 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.547393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.547394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.547521 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.547756 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.547757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.547757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.548019 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:45.548236 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:45.548324 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:24:45.548482 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260926 09:24:45.577231 6815 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:24:45.577814 6815 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:45.589869 6815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Generated new TSK 0
I20260926 09:24:45.590106 6815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:24:45.627879 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:45.711762 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:45.711972 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:45.711998 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:45.713267 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:45.713311 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:45.713338 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:24:45.714825 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:45.715075 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:45.715318 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:45.720381 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:45.720409 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:45.720492 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:45.720988 6819 server_base.cc:1034] running on GCE node
I20260926 09:24:45.721248 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:45.721493 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:45.722671 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790414685722659 us; error 38 us; skew 500 ppm
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:45.725283 6819 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:24:45.725751 6819 webserver.cc:460] Webserver started at http://127.24.235.65:33765/ using document root <none> and password file <none>
I20260926 09:24:45.725989 6819 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:45.726038 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:45.726171 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:45.727222 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "9ae3430798ff44e19ba3a567044e08d5"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "a74b712a764ff61d057458845168fc18"
server_key_iv: "ce65e01b4b6414ba6a9af483ccde947e"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.727584 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "9ae3430798ff44e19ba3a567044e08d5"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "a74b712a764ff61d057458845168fc18"
server_key_iv: "ce65e01b4b6414ba6a9af483ccde947e"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.729132 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:24:45.729951 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.730135 6819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:45.730228 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "9ae3430798ff44e19ba3a567044e08d5"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "a74b712a764ff61d057458845168fc18"
server_key_iv: "ce65e01b4b6414ba6a9af483ccde947e"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.730286 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:45.738718 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:45.740834 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:45.740943 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:45.741186 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:45.741487 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:45.741519 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.741540 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:45.741585 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.748173 6819 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46489
I20260926 09:24:45.748281 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46489 every 8 connection(s)
I20260926 09:24:45.748624 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:24:45.753497 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6819
I20260926 09:24:45.753693 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:24:45.753894 25517 external_mini_cluster.cc:1428] Setting key 8d615b005c65dc372f5e72ae7b42d632
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:45.759616 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:45.749456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:39699 (local address 127.24.235.126:37637)
0926 09:24:45.749510 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:45.749512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:45.749727 (+ 215us) server_negotiation.cc:407] Connection header received
0926 09:24:45.750310 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:45.750311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:45.750325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:45.750388 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:45.751501 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.752066 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.753242 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.753473 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.753561 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:45.756139 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:45.756141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:45.756141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:45.756152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:45.757905 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.758484 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.758485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.758486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.758578 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.758891 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.758892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.758892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.759022 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:45.759105 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:45.759404 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 09:24:45.759502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:24:45.759825 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:45.749412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:39699)
0926 09:24:45.749595 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:45.749632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:45.750199 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:45.750476 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:45.750481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:45.751187 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:45.751389 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:45.751394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.752173 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.752174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:45.753149 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:45.753151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.753570 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.753572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:45.753728 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:45.754368 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:45.754376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:45.756035 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:45.758006 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:45.758014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:45.758017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:45.758394 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:45.758670 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:45.758671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:45.758672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:45.758797 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:45.759096 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:45.759104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:45.759251 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 09:24:45.759379 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 09:24:45.759559 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260926 09:24:45.760456 6948 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:45.760603 6948 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:45.760804 6948 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:45.761443 6757 ts_manager.cc:194] Registered new tserver with Master: 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489)
I20260926 09:24:45.762504 6757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:39699
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:24:45.782177 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:45.866601 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:45.866807 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:45.866830 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:45.867904 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:45.867938 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:45.867964 6953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:24:45.869863 6953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:45.870102 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:45.870366 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:45.876173 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:45.876173 6953 server_base.cc:1034] running on GCE node
W20260926 09:24:45.876200 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:45.876173 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:45.876626 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:45.876930 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:45.878087 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790414685878075 us; error 32 us; skew 500 ppm
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:45.880685 6953 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:24:45.881227 6953 webserver.cc:460] Webserver started at http://127.24.235.66:43961/ using document root <none> and password file <none>
I20260926 09:24:45.881420 6953 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:45.881471 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:45.881567 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:45.882588 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "0b17396fb9d3da94dc15092b68e8a229"
server_key_iv: "f70e476e1f6ae12d27437726a99a8088"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.882901 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "0b17396fb9d3da94dc15092b68e8a229"
server_key_iv: "f70e476e1f6ae12d27437726a99a8088"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.884402 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:24:45.885236 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.885438 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:45.885555 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "0b17396fb9d3da94dc15092b68e8a229"
server_key_iv: "f70e476e1f6ae12d27437726a99a8088"
server_key_version: "encryptionkey@0"
I20260926 09:24:45.885643 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:45.908860 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:45.911059 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:45.911278 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:45.911561 6953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:45.911952 6953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:45.912006 6953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.912047 6953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:45.912084 6953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:45.918829 6953 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33439
I20260926 09:24:45.918943 7081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33439 every 8 connection(s)
I20260926 09:24:45.919262 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
Sep 26 09:24:45 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414685, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:45.928812 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:45.920129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:56905 (local address 127.24.235.126:37637)
0926 09:24:45.920193 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:45.920195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:45.920390 (+ 195us) server_negotiation.cc:407] Connection header received
0926 09:24:45.920946 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:45.920947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:45.920959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:45.921094 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:45.922070 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.922594 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.923580 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.923787 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.923844 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:45.925939 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:45.925940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:45.925941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:45.925947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:45.927432 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.927869 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.927870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.927871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.927941 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:45.928190 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:45.928191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:45.928191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:45.928317 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:45.928373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:45.928646 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 09:24:45.928747 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:24:45.928887 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:45.920084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:56905)
0926 09:24:45.920237 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:45.920280 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:45.920832 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:45.921167 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:45.921171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:45.921770 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:45.921966 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:45.921970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.922689 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.922690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:45.923499 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:45.923501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:45.923869 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:45.923870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:45.923990 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:45.924480 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:45.924488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:45.925843 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:45.927517 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:45.927524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:45.927526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:45.927777 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:45.928012 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:45.928013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:45.928013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:45.928114 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:45.928377 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:45.928382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:45.928499 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:24:45.928583 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:24:45.928710 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260926 09:24:45.929200 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 6953
I20260926 09:24:45.929307 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:24:45.929550 25517 external_mini_cluster.cc:1428] Setting key 213d134593f9f0bef63f230142c28803
I20260926 09:24:45.929545 7082 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:45.929652 7082 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:45.929848 7082 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:45.930264 6757 ts_manager.cc:194] Registered new tserver with Master: 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
I20260926 09:24:45.930639 6757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:56905
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:24:45.956611 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:46.042799 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:46.042999 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:46.043022 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:46.044198 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:46.044253 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:46.044291 7087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:24:46.045866 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:46.046096 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:46.046320 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:46.051837 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.051833 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.051921 7087 server_base.cc:1034] running on GCE node
W20260926 09:24:46.051815 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.052177 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:46.052452 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:46.053591 7087 hybrid_clock.cc:648] HybridClock initialized: now 1790414686053565 us; error 41 us; skew 500 ppm
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:46.056150 7087 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:24:46.056612 7087 webserver.cc:460] Webserver started at http://127.24.235.67:35175/ using document root <none> and password file <none>
I20260926 09:24:46.056864 7087 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:46.056907 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:46.057128 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:24:46.058202 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "793a58d1c50a4be4a75a1909c505442a"
format_stamp: "Formatted at 2026-09-26 09:24:46 on dist-test-slave-7zlw"
server_key: "25ba1a56f26f9f201830e769dc66ecc1"
server_key_iv: "a197d75078220fde125a743593ee9756"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.058554 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "793a58d1c50a4be4a75a1909c505442a"
format_stamp: "Formatted at 2026-09-26 09:24:46 on dist-test-slave-7zlw"
server_key: "25ba1a56f26f9f201830e769dc66ecc1"
server_key_iv: "a197d75078220fde125a743593ee9756"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.059864 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:24:46.060595 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.060801 7087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:24:46.060880 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "793a58d1c50a4be4a75a1909c505442a"
format_stamp: "Formatted at 2026-09-26 09:24:46 on dist-test-slave-7zlw"
server_key: "25ba1a56f26f9f201830e769dc66ecc1"
server_key_iv: "a197d75078220fde125a743593ee9756"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.060935 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:46.072772 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:46.075151 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:46.075268 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:46.075461 7087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:46.075742 7087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:46.075773 7087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.075794 7087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:46.075806 7087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.081947 7087 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:42935
I20260926 09:24:46.082000 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:42935 every 8 connection(s)
I20260926 09:24:46.082307 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:46.091617 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.083054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:60279 (local address 127.24.235.126:37637)
0926 09:24:46.083102 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.083104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.083316 (+ 212us) server_negotiation.cc:407] Connection header received
0926 09:24:46.083782 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.083783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.083794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.083856 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.084771 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.085343 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.086369 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.086558 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.086625 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.088759 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.088760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.088760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.088766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.090459 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.090832 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.090832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.090833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.090882 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.091083 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.091083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.091084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.091191 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.091241 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.091484 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.091546 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:24:46.091782 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.082984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:60279)
0926 09:24:46.083155 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.083198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.083693 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.083908 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.083911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.084465 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:46.084666 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.084670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.085449 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.085449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.086263 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.086265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.086660 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.086661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.086765 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.087229 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:46.087237 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:46.088684 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:46.090535 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.090541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.090543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.090767 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.090938 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.090939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.090939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.091029 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.091245 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:46.091249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:46.091356 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.091435 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.091548 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 09:24:46.092046 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7087
I20260926 09:24:46.092132 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:24:46.092340 25517 external_mini_cluster.cc:1428] Setting key 0f90307cd845b50a321acd43f64cc6eb
I20260926 09:24:46.092398 7216 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:46.092499 7216 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:46.092702 7216 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:46.093122 6756 ts_manager.cc:194] Registered new tserver with Master: 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935)
I20260926 09:24:46.093516 6756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:60279
I20260926 09:24:46.104331 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:24:46.112371 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.105835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.24.235.126:37637)
0926 09:24:46.105901 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.105904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.106005 (+ 101us) server_negotiation.cc:407] Connection header received
0926 09:24:46.106076 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.106077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.106090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.106132 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.106893 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.107421 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.108138 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.108329 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.108382 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.109596 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.109598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.109598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.109605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.111209 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.111567 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.111568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.111568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.111622 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.111835 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.111836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.111836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.111981 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.112082 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.112210 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.112272 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:24:46.114580 6756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:24:46.116190 6756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:24:46.123541 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.119107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.24.235.65:46489)
0926 09:24:46.119355 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.119359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.119674 (+ 315us) server_negotiation.cc:407] Connection header received
0926 09:24:46.119720 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.119722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.119738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.119808 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.120063 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.120790 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.122359 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123102 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123165 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123227 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.123408 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.123457 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260926 09:24:46.123538 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.118844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33439 (local address 127.0.0.1:56540)
0926 09:24:46.119556 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.119565 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.119605 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.119839 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.119840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.119842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.119944 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.119945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.121326 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.121327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.122258 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.122261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123204 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.123304 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123314 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.123392 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.123436 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":51,"threads_started":1}
I20260926 09:24:46.123544 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.118840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:42935 (local address 127.0.0.1:44802)
0926 09:24:46.119000 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.119042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.119106 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.119511 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.119514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.119516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.119591 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.119593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.121162 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.121163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.122255 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.122258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123193 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.123280 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123296 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.123345 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.123400 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":9,"thread_start_us":107,"threads_started":1}
I20260926 09:24:46.123720 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.119096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.24.235.67:42935)
0926 09:24:46.119333 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.119336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.119346 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:46.119389 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.119390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.119403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.119491 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.119895 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.121064 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.122359 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123090 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123473 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123538 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.123565 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.123623 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260926 09:24:46.124106 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.118843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.0.0.1:45830)
0926 09:24:46.119398 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.119407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.119458 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.119941 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.119942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.119944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.119994 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.119995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.120896 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.120897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.122255 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.122258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123257 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.123305 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123308 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.123372 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.123401 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"spinlock_wait_cycles":768,"thread_start_us":113,"threads_started":1}
I20260926 09:24:46.124449 7150 tablet_service.cc:1467] Processing CreateTablet for tablet 3505e62ac0d447e68fafc34db56c624d (DEFAULT_TABLE table=test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.124833 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3505e62ac0d447e68fafc34db56c624d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.124864 6882 tablet_service.cc:1467] Processing CreateTablet for tablet 3505e62ac0d447e68fafc34db56c624d (DEFAULT_TABLE table=test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.125211 6882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3505e62ac0d447e68fafc34db56c624d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.123929 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.118947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.24.235.66:33439)
0926 09:24:46.119283 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.119288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.119668 (+ 380us) server_negotiation.cc:407] Connection header received
0926 09:24:46.119738 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.119741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.119758 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.119841 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.120007 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.121106 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.122355 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.123083 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.123202 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.123262 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.123758 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.123836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":84,"threads_started":1}
I20260926 09:24:46.128422 7234 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Bootstrap starting.
I20260926 09:24:46.128986 7016 tablet_service.cc:1467] Processing CreateTablet for tablet 3505e62ac0d447e68fafc34db56c624d (DEFAULT_TABLE table=test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.129194 7233 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap starting.
I20260926 09:24:46.129242 7234 tablet_bootstrap.cc:654] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.129384 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3505e62ac0d447e68fafc34db56c624d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.129875 7234 log.cc:824] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.130033 7233 tablet_bootstrap.cc:654] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.130702 7233 log.cc:824] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.130949 7234 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: No bootstrap required, opened a new log
I20260926 09:24:46.131065 7234 ts_tablet_manager.cc:1397] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:24:46.131528 7233 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: No bootstrap required, opened a new log
I20260926 09:24:46.131603 7233 ts_tablet_manager.cc:1397] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:24:46.132346 7237 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Bootstrap starting.
I20260926 09:24:46.132972 7237 tablet_bootstrap.cc:654] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.133329 7234 raft_consensus.cc:348] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.133481 7237 log.cc:824] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.133513 7234 raft_consensus.cc:374] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.133571 7234 raft_consensus.cc:729] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a58d1c50a4be4a75a1909c505442a, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.133699 7234 consensus_queue.cc:260] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.133917 7233 raft_consensus.cc:348] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.134063 7234 ts_tablet_manager.cc:1428] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:24:46.134084 7233 raft_consensus.cc:374] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.134121 7233 raft_consensus.cc:729] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3430798ff44e19ba3a567044e08d5, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.134076 7216 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
I20260926 09:24:46.134318 7237 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: No bootstrap required, opened a new log
I20260926 09:24:46.134274 7233 consensus_queue.cc:260] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.134371 7237 ts_tablet_manager.cc:1397] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:24:46.134588 7233 ts_tablet_manager.cc:1428] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:24:46.134809 6948 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
I20260926 09:24:46.136262 7237 raft_consensus.cc:348] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.136412 7237 raft_consensus.cc:374] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.136457 7237 raft_consensus.cc:729] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418f3551b43646b6b0bc9c4f44ac3f53, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.136566 7237 consensus_queue.cc:260] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.136832 7237 ts_tablet_manager.cc:1428] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 09:24:46.136912 7082 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
W20260926 09:24:46.169895 7083 tablet.cc:2369] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:24:46.230403 7241 raft_consensus.cc:482] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:46.230563 7241 raft_consensus.cc:504] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.230931 7241 leader_election.cc:288] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489), 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935)
I20260926 09:24:46.234058 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.231190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:42935 (local address 127.24.235.66:59171)
0926 09:24:46.231282 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.231296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.231360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.231559 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.231560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.231562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.231637 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.231639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.232302 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.232303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.233179 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.233182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.233771 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.233772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.233866 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.233870 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.233925 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.233965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 09:24:46.234275 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.231277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:59171 (local address 127.24.235.67:42935)
0926 09:24:46.231334 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.231336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.231346 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:46.231435 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.231435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.231445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.231514 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.231727 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.232207 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.233277 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.233686 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.233773 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.233782 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.234166 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.234217 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:24:46.234737 7170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3505e62ac0d447e68fafc34db56c624d" candidate_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a58d1c50a4be4a75a1909c505442a" is_pre_election: true
I20260926 09:24:46.234926 7170 raft_consensus.cc:2454] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418f3551b43646b6b0bc9c4f44ac3f53 in term 0.
I20260926 09:24:46.234941 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.231315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:52045 (local address 127.24.235.65:46489)
0926 09:24:46.231379 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.231381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.231642 (+ 261us) server_negotiation.cc:407] Connection header received
0926 09:24:46.231758 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.231759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.231774 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.231856 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.232029 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.232648 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.233940 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.234555 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.234648 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.234657 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.234834 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.234875 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:24:46.234961 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.231192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.24.235.66:52045)
0926 09:24:46.231403 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.231463 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.231663 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.231860 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.231861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.231862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.231953 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.231954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.232739 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.232740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.233830 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.233834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.234647 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.234648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.234743 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.234747 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.234814 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.234857 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":30,"thread_start_us":102,"threads_started":1}
I20260926 09:24:46.235297 6902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3505e62ac0d447e68fafc34db56c624d" candidate_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3430798ff44e19ba3a567044e08d5" is_pre_election: true
I20260926 09:24:46.235458 6902 raft_consensus.cc:2454] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418f3551b43646b6b0bc9c4f44ac3f53 in term 0.
I20260926 09:24:46.235659 6971 leader_election.cc:302] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a; no voters:
I20260926 09:24:46.235827 7241 raft_consensus.cc:2781] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:24:46.235927 7241 raft_consensus.cc:482] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:24:46.235965 7241 raft_consensus.cc:3037] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.236763 7241 raft_consensus.cc:504] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.236928 7241 leader_election.cc:288] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [CANDIDATE]: Term 1 election: Requested vote from peers 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489), 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935)
I20260926 09:24:46.237112 7170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3505e62ac0d447e68fafc34db56c624d" candidate_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a58d1c50a4be4a75a1909c505442a"
I20260926 09:24:46.237130 6902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3505e62ac0d447e68fafc34db56c624d" candidate_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3430798ff44e19ba3a567044e08d5"
I20260926 09:24:46.237215 6902 raft_consensus.cc:3037] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.237221 7170 raft_consensus.cc:3037] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.237990 6902 raft_consensus.cc:2454] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418f3551b43646b6b0bc9c4f44ac3f53 in term 1.
I20260926 09:24:46.238089 7170 raft_consensus.cc:2454] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418f3551b43646b6b0bc9c4f44ac3f53 in term 1.
I20260926 09:24:46.238168 6972 leader_election.cc:302] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f3551b43646b6b0bc9c4f44ac3f53, 9ae3430798ff44e19ba3a567044e08d5; no voters:
I20260926 09:24:46.238276 7241 raft_consensus.cc:2781] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:46.238448 7241 raft_consensus.cc:686] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 LEADER]: Becoming Leader. State: Replica: 418f3551b43646b6b0bc9c4f44ac3f53, State: Running, Role: LEADER
I20260926 09:24:46.238551 7241 consensus_queue.cc:237] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.239418 6756 catalog_manager.cc:5495] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } health_report { overall_health: UNKNOWN } } }
W20260926 09:24:46.249531 6949 tablet.cc:2369] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:24:46.267737 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.265380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.24.235.66:33439)
0926 09:24:46.265435 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.265438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.265447 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:24:46.265494 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.265495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.265514 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.265591 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:24:46.265728 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.266199 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.266966 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.267161 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.267235 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.267371 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:24:46.267478 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:24:46.267564 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.267605 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.267627 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:24:46.269714 7170 raft_consensus.cc:1261] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Refusing update from remote peer 418f3551b43646b6b0bc9c4f44ac3f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:46.269714 6902 raft_consensus.cc:1261] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Refusing update from remote peer 418f3551b43646b6b0bc9c4f44ac3f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:46.270098 7241 consensus_queue.cc:1035] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:46.270258 7243 consensus_queue.cc:1035] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:46.272397 7246 mvcc.cc:205] Tried to move back new op lower bound from 7333538554957619200 to 7333538554833813504. Current Snapshot: MvccSnapshot[applied={T|T < 7333538554957619200}]
I20260926 09:24:46.272827 7247 mvcc.cc:205] Tried to move back new op lower bound from 7333538554957619200 to 7333538554833813504. Current Snapshot: MvccSnapshot[applied={T|T < 7333538554957619200}]
I20260926 09:24:46.273646 7248 mvcc.cc:205] Tried to move back new op lower bound from 7333538554957619200 to 7333538554833813504. Current Snapshot: MvccSnapshot[applied={T|T < 7333538554957619200}]
I20260926 09:24:46.274770 6756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:46.274848 6756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:46.275645 6756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Sending DeleteTablet for 3 replicas of tablet 3505e62ac0d447e68fafc34db56c624d
I20260926 09:24:46.275964 7016 tablet_service.cc:1514] Processing DeleteTablet for tablet 3505e62ac0d447e68fafc34db56c624d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:56540
I20260926 09:24:46.276202 6882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3505e62ac0d447e68fafc34db56c624d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:45830
I20260926 09:24:46.276414 7256 tablet_replica.cc:330] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: stopping tablet replica
I20260926 09:24:46.276424 7150 tablet_service.cc:1514] Processing DeleteTablet for tablet 3505e62ac0d447e68fafc34db56c624d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:44802
I20260926 09:24:46.276520 7256 raft_consensus.cc:2229] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:24:46.276634 7256 raft_consensus.cc:2258] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.276652 7257 tablet_replica.cc:330] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: stopping tablet replica
I20260926 09:24:46.276815 7255 tablet_replica.cc:330] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: stopping tablet replica
I20260926 09:24:46.276820 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6819
I20260926 09:24:46.277901 7255 raft_consensus.cc:2229] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:24:46.278026 7255 raft_consensus.cc:2258] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.278478 7255 ts_tablet_manager.cc:1905] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:46.280273 7255 ts_tablet_manager.cc:1918] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:46.280336 7255 log.cc:1196] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/3505e62ac0d447e68fafc34db56c624d
I20260926 09:24:46.280610 7255 ts_tablet_manager.cc:1939] T 3505e62ac0d447e68fafc34db56c624d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting consensus metadata
I20260926 09:24:46.281129 6741 catalog_manager.cc:4841] TS 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): tablet 3505e62ac0d447e68fafc34db56c624d (table test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]) successfully deleted
W20260926 09:24:46.283609 6744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:46489 (error 108)
I20260926 09:24:46.283890 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6953
I20260926 09:24:46.284049 7257 raft_consensus.cc:2229] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:24:46.284118 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.283991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.0.0.1:45832)
0926 09:24:46.284038 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.284071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:24:46.284230 6744 catalog_manager.cc:4584] TS 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3505e62ac0d447e68fafc34db56c624d: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46489: connect: Connection refused (error 111)
I20260926 09:24:46.284339 7257 raft_consensus.cc:2258] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.285774 7257 ts_tablet_manager.cc:1905] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:46.287484 7257 ts_tablet_manager.cc:1918] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:46.287561 7257 log.cc:1196] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/3505e62ac0d447e68fafc34db56c624d
I20260926 09:24:46.287844 7257 ts_tablet_manager.cc:1939] T 3505e62ac0d447e68fafc34db56c624d P 793a58d1c50a4be4a75a1909c505442a: Deleting consensus metadata
I20260926 09:24:46.288338 6743 catalog_manager.cc:4841] TS 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): tablet 3505e62ac0d447e68fafc34db56c624d (table test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]) successfully deleted
I20260926 09:24:46.289876 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7087
I20260926 09:24:46.294807 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 6725
I20260926 09:24:46.300493 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--webserver_interface=127.24.235.126
--webserver_port=35479
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:46.384030 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:46.384253 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:46.384276 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:46.385507 7259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:46.385557 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:46.385572 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:46.385586 7259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:46.385597 7259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:46.387089 7259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:46.387344 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:46.387563 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:46.397974 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.397974 7265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.398098 7268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.398254 7259 server_base.cc:1034] running on GCE node
I20260926 09:24:46.398561 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:46.398878 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:46.400046 7259 hybrid_clock.cc:648] HybridClock initialized: now 1790414686400019 us; error 38 us; skew 500 ppm
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:46.402858 7259 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:46.403434 7259 webserver.cc:460] Webserver started at http://127.24.235.126:35479/ using document root <none> and password file <none>
I20260926 09:24:46.403709 7259 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:46.403798 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:46.405368 7259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:24:46.406073 7275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.406316 7259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:24:46.406430 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "4f4314861bb847899b76b29f48154992"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "d6a84c46b8d7373284922719654fba4b"
server_key_iv: "8a3b3dd11c40d93867cc6f8527619825"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.406917 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:46.420367 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:46.422822 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:46.422987 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:46.427594 7259 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37637
I20260926 09:24:46.427626 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37637 every 8 connection(s)
I20260926 09:24:46.428027 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:46.429759 7328 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:24:46.430351 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap starting.
I20260926 09:24:46.436518 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7259
I20260926 09:24:46.436857 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:46489
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=33765
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:46.439556 7328 log.cc:824] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.441795 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:24:46.442114 7328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap complete.
I20260926 09:24:46.444519 7328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:46.444777 7328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4314861bb847899b76b29f48154992, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.444941 7328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:46.445004 7328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:46.445096 7328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:46.445142 7328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:24:46.446200 7328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:46.446286 7328 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4314861bb847899b76b29f48154992; no voters:
I20260926 09:24:46.446496 7328 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:24:46.446532 7332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:24:46.446635 7332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 LEADER]: Becoming Leader. State: Replica: 4f4314861bb847899b76b29f48154992, State: Running, Role: LEADER
I20260926 09:24:46.446727 7332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:46.446767 7328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:46.446983 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:46.447074 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:46.447196 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4314861bb847899b76b29f48154992. Latest consensus state: current_term: 2 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:46.447276 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:46.447645 7336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:46.448572 7336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:46.448683 7336 catalog_manager.cc:1259] Loaded cluster ID: 55fce923d21a45b69f4bdf846f8bfd2a
I20260926 09:24:46.448726 7336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:24:46.449167 7336 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 09:24:46.451930 7336 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 09:24:46.458181 7336 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:24:46.458739 7336 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Loaded TSK: 0
I20260926 09:24:46.458976 7336 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:24:46.539494 7330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:46.539701 7330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:46.539724 7330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:46.540902 7330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:46.540943 7330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:46.540970 7330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:24:46.542440 7330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:46489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=33765
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:46.542714 7330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:46.542955 7330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:46.549053 7358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.549103 7356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.549098 7355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.549767 7330 server_base.cc:1034] running on GCE node
I20260926 09:24:46.550004 7330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:46.550303 7330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:46.551481 7330 hybrid_clock.cc:648] HybridClock initialized: now 1790414686551456 us; error 40 us; skew 500 ppm
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:46.554409 7330 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:24:46.554970 7330 webserver.cc:460] Webserver started at http://127.24.235.65:33765/ using document root <none> and password file <none>
I20260926 09:24:46.555236 7330 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:46.555297 7330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:46.556804 7330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:24:46.557626 7365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.557806 7330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:24:46.557921 7330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "9ae3430798ff44e19ba3a567044e08d5"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "a74b712a764ff61d057458845168fc18"
server_key_iv: "ce65e01b4b6414ba6a9af483ccde947e"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.558291 7330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:46.576685 7330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:46.579164 7330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:46.579340 7330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:46.579751 7330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:46.580307 7372 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:24:46.581949 7330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:24:46.582011 7330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:24:46.582041 7330 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:24:46.582733 7330 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:24:46.582782 7330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:24:46.582824 7372 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap starting.
I20260926 09:24:46.591714 7330 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46489
I20260926 09:24:46.591888 7479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46489 every 8 connection(s)
I20260926 09:24:46.592265 7330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:24:46.593482 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7330
I20260926 09:24:46.593766 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:33439
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=43961
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:46.595342 7372 log.cc:824] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.599056 7372 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:24:46.599519 7372 tablet_bootstrap.cc:492] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap complete.
I20260926 09:24:46.600965 7372 ts_tablet_manager.cc:1397] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260926 09:24:46.602769 7372 raft_consensus.cc:348] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.603237 7372 raft_consensus.cc:729] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3430798ff44e19ba3a567044e08d5, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.603529 7372 consensus_queue.cc:260] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } }
I20260926 09:24:46.603999 7372 ts_tablet_manager.cc:1428] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:46.608771 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.593070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:46643)
0926 09:24:46.595429 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.595482 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.596134 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.597191 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.597196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.597590 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:46.597852 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.597859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.599565 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.599566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.601005 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.601008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.601890 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.601892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.602066 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.602725 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:46.602736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:46.604618 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:46.606935 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.606944 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.606946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.607196 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.607527 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.607528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.607528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.607650 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.608069 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:46.608076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:46.608221 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.608361 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.608549 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2307,"thread_start_us":101,"threads_started":1}
I20260926 09:24:46.609727 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.593235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:46643 (local address 127.24.235.126:37637)
0926 09:24:46.596665 (+ 3430us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.596674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.596721 (+ 47us) server_negotiation.cc:407] Connection header received
0926 09:24:46.596877 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.596886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.596958 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.597156 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.598035 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.599406 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.601179 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.601748 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.602393 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.604753 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.604758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.604759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.604772 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.606818 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.607310 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.607313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.607313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.607427 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.607726 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.607727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.607728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.607973 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.608093 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.608419 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.608668 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3373,"thread_start_us":99,"threads_started":1}
I20260926 09:24:46.610364 7480 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:46.610489 7480 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:46.610757 7480 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:46.611512 7292 ts_manager.cc:194] Registered new tserver with Master: 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489)
I20260926 09:24:46.612638 7292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:46643
I20260926 09:24:46.613770 7480 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
I20260926 09:24:46.617373 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.612155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.0.0.1:45848)
0926 09:24:46.613278 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.613318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.613403 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.613771 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.613774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.613776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.613848 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.613850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.615254 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.615255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.616121 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.616124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.617100 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.617102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.617199 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.617209 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.617279 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.617321 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":102,"threads_started":1}
I20260926 09:24:46.617425 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.612948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.24.235.65:46489)
0926 09:24:46.613140 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.613144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.613402 (+ 258us) server_negotiation.cc:407] Connection header received
0926 09:24:46.613490 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.613492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.613651 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.613742 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.614024 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.615155 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.616221 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.616965 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.617064 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.617118 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.617314 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.617356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260926 09:24:46.617861 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 3505e62ac0d447e68fafc34db56c624d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:45848
I20260926 09:24:46.618392 7489 tablet_replica.cc:330] stopping tablet replica
I20260926 09:24:46.618503 7489 raft_consensus.cc:2229] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:24:46.618572 7489 raft_consensus.cc:2258] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.618914 7489 ts_tablet_manager.cc:1905] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:46.621505 7489 ts_tablet_manager.cc:1918] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:46.621585 7489 log.cc:1196] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/3505e62ac0d447e68fafc34db56c624d
I20260926 09:24:46.621819 7489 ts_tablet_manager.cc:1939] T 3505e62ac0d447e68fafc34db56c624d P 9ae3430798ff44e19ba3a567044e08d5: Deleting consensus metadata
I20260926 09:24:46.622318 7279 catalog_manager.cc:4841] TS 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489): tablet 3505e62ac0d447e68fafc34db56c624d (table test-table [id=3ba04a379b604e71bfdbde14b95e6ed9]) successfully deleted
W20260926 09:24:46.684304 7484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:46.684520 7484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:46.684542 7484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:46.685778 7484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:46.685823 7484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:46.685850 7484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:24:46.687605 7484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=43961
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:46.687867 7484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:46.688120 7484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:46.693333 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.693398 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.693346 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.693569 7484 server_base.cc:1034] running on GCE node
I20260926 09:24:46.693862 7484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:46.694111 7484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:46.695282 7484 hybrid_clock.cc:648] HybridClock initialized: now 1790414686695257 us; error 44 us; skew 500 ppm
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:46.698083 7484 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:24:46.698596 7484 webserver.cc:460] Webserver started at http://127.24.235.66:43961/ using document root <none> and password file <none>
I20260926 09:24:46.698820 7484 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:46.698904 7484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:46.700380 7484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:46.701149 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.701375 7484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:46.701476 7484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "0b17396fb9d3da94dc15092b68e8a229"
server_key_iv: "f70e476e1f6ae12d27437726a99a8088"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.701887 7484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:46.716315 7484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:46.718497 7484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:46.718668 7484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:46.718909 7484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:46.719223 7484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:46.719282 7484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.719324 7484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:46.719349 7484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.725580 7484 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33439
I20260926 09:24:46.725623 7617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33439 every 8 connection(s)
I20260926 09:24:46.725957 7484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:24:46.730885 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7484
I20260926 09:24:46.731194 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:42935
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=35175
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:46.736367 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.726677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:33473 (local address 127.24.235.126:37637)
0926 09:24:46.726722 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.726724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.726977 (+ 253us) server_negotiation.cc:407] Connection header received
0926 09:24:46.727512 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.727513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.727525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.727585 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.728576 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.729205 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.730198 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.730418 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.730524 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.733145 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.733146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.733147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.733153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.734809 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.735200 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.735201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.735202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.735268 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.735497 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.735498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.735499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.735711 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.735799 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.736118 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.736220 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:24:46.736399 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.726679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:33473)
0926 09:24:46.726856 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.726890 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.727409 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.727693 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.727696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.728276 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:46.728465 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.728469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.729297 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.729298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.730095 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.730097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.730533 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.730534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.730662 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.731205 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:46.731214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:46.733056 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:46.734892 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.734898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.734900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.735116 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.735338 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.735339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.735339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.735434 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.735814 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:46.735821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:46.735950 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.736077 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.736217 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260926 09:24:46.736970 7618 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:46.737116 7618 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:46.737336 7618 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:46.737810 7292 ts_manager.cc:194] Registered new tserver with Master: 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
I20260926 09:24:46.738353 7292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:33473
W20260926 09:24:46.815445 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:46.815647 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:46.815671 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:46.816845 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:46.816892 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:46.816918 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:24:46.818395 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:42935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=35175
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:46.818673 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:46.818915 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:46.824062 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:46.824069 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.824172 7621 server_base.cc:1034] running on GCE node
W20260926 09:24:46.824079 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:46.824493 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:46.824716 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:46.825860 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790414686825847 us; error 28 us; skew 500 ppm
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:46.828617 7621 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:24:46.829159 7621 webserver.cc:460] Webserver started at http://127.24.235.67:35175/ using document root <none> and password file <none>
I20260926 09:24:46.829414 7621 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:46.829507 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:46.830914 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:46.831533 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.831743 7621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:46.831847 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "793a58d1c50a4be4a75a1909c505442a"
format_stamp: "Formatted at 2026-09-26 09:24:46 on dist-test-slave-7zlw"
server_key: "25ba1a56f26f9f201830e769dc66ecc1"
server_key_iv: "a197d75078220fde125a743593ee9756"
server_key_version: "encryptionkey@0"
I20260926 09:24:46.832213 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:46.839895 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:46.842024 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:46.842190 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:46.842417 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:46.842733 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:46.842788 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.842831 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:46.842859 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.849473 7621 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:42935
I20260926 09:24:46.849517 7749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:42935 every 8 connection(s)
I20260926 09:24:46.849869 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
Sep 26 09:24:46 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414686, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:24:46.857458 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7621
I20260926 09:24:46.859277 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.850618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:49523 (local address 127.24.235.126:37637)
0926 09:24:46.850687 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.850689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.850883 (+ 194us) server_negotiation.cc:407] Connection header received
0926 09:24:46.851376 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.851377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.851391 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.851451 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.852401 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.852891 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.853994 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.854192 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.854246 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.856284 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.856286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.856287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.856294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.857789 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.858253 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.858254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.858254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.858307 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.858627 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.858627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.858627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.858731 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.858780 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.859101 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.859182 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:24:46.859361 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.850577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:49523)
0926 09:24:46.850747 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.850791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.851285 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.851504 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.851507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.852102 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:46.852304 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.852308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.852978 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.852979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.853920 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.853922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.854269 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.854269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.854374 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.854833 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:46.854840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:46.856187 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:46.857867 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.857874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.857876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.858184 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.858373 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:46.858374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:46.858374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:46.858518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:46.858801 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:46.858807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:46.858972 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.859055 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.859171 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260926 09:24:46.859804 7750 heartbeater.cc:347] Connected to a master server at 127.24.235.126:37637
I20260926 09:24:46.859896 7750 heartbeater.cc:464] Registering TS with master...
I20260926 09:24:46.860081 7750 heartbeater.cc:511] Master 127.24.235.126:37637 requested a full tablet report, sending...
I20260926 09:24:46.860455 7292 ts_manager.cc:194] Registered new tserver with Master: 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935)
I20260926 09:24:46.860824 7292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:49523
I20260926 09:24:46.864557 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.857886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.24.235.126:37637)
0926 09:24:46.857990 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.857992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.858025 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:24:46.858113 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.858114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.858123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.858177 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:46.858890 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.859577 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.860390 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.860635 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.860707 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.861992 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:46.861994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:46.861994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:46.862001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:46.863393 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.863753 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.863754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.863754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.863815 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:46.864019 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:46.864019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:46.864020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:46.864144 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:46.864340 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.864376 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.864453 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260926 09:24:46.865337 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:24:46.867810 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.865663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.24.235.126:37637)
0926 09:24:46.865715 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.865717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.865771 (+ 54us) server_negotiation.cc:407] Connection header received
0926 09:24:46.865830 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.865831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.865860 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.865920 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:24:46.865990 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.866445 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.867154 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.867338 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.867389 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.867537 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:24:46.867632 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:24:46.867690 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.867748 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.867764 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:24:46.868063 7292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:24:46.869858 7292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:24:46.872272 7414 tablet_service.cc:1467] Processing CreateTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d (DEFAULT_TABLE table=test-table [id=b114447cee4643d38d56f3eeac63e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.872473 7414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357c4cc7bc84ecc8e346037b1b2b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.874328 7372 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap starting.
I20260926 09:24:46.874979 7372 tablet_bootstrap.cc:654] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.877305 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.872028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33439 (local address 127.0.0.1:56560)
0926 09:24:46.872194 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.872205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.872259 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.872677 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.872679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.872682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.872750 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.872752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.874556 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.874557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.875766 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.875770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.877075 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.877076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.877156 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.877162 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.877197 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.877243 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260926 09:24:46.877306 7372 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: No bootstrap required, opened a new log
I20260926 09:24:46.877394 7372 ts_tablet_manager.cc:1397] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 09:24:46.877511 7372 raft_consensus.cc:348] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.877571 7372 raft_consensus.cc:374] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.877606 7372 raft_consensus.cc:729] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3430798ff44e19ba3a567044e08d5, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.877668 7372 consensus_queue.cc:260] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.877769 7372 ts_tablet_manager.cc:1428] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:46.877790 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.872297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.24.235.66:33439)
0926 09:24:46.872487 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.872493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.872504 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:24:46.872548 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.872550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.872570 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.872661 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.872861 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.874263 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.875886 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.876755 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.877523 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.877611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.877642 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.877694 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
I20260926 09:24:46.878540 7552 tablet_service.cc:1467] Processing CreateTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d (DEFAULT_TABLE table=test-table [id=b114447cee4643d38d56f3eeac63e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.878834 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.871891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:42935 (local address 127.0.0.1:44810)
0926 09:24:46.871936 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.871947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.871984 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.874591 (+ 2607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.874593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.874594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.874640 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.874642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.876063 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.876064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.877295 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.877298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.878611 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.878612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.878681 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.878684 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.878737 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.878780 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 09:24:46.878901 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357c4cc7bc84ecc8e346037b1b2b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.879074 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.871993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.24.235.67:42935)
0926 09:24:46.874399 (+ 2406us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.874404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.874417 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:24:46.874460 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.874462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.874479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.874722 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.874794 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.875938 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.877961 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.878505 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.878607 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.878663 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.878948 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.878997 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2357,"thread_start_us":101,"threads_started":1}
I20260926 09:24:46.881928 7684 tablet_service.cc:1467] Processing CreateTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d (DEFAULT_TABLE table=test-table [id=b114447cee4643d38d56f3eeac63e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:24:46.882323 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357c4cc7bc84ecc8e346037b1b2b21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:24:46.882488 7760 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Bootstrap starting.
I20260926 09:24:46.883260 7760 tablet_bootstrap.cc:654] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.883715 7760 log.cc:824] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.884400 7760 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: No bootstrap required, opened a new log
I20260926 09:24:46.884455 7760 ts_tablet_manager.cc:1397] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:24:46.885556 7762 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Bootstrap starting.
I20260926 09:24:46.886240 7762 tablet_bootstrap.cc:654] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Neither blocks nor log segments found. Creating new log.
I20260926 09:24:46.886173 7760 raft_consensus.cc:348] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.886353 7760 raft_consensus.cc:374] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.886404 7760 raft_consensus.cc:729] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418f3551b43646b6b0bc9c4f44ac3f53, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.886539 7760 consensus_queue.cc:260] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.886854 7618 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
I20260926 09:24:46.886876 7762 log.cc:824] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:46.887145 7760 ts_tablet_manager.cc:1428] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:24:46.887696 7762 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: No bootstrap required, opened a new log
I20260926 09:24:46.887781 7762 ts_tablet_manager.cc:1397] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:24:46.889279 7762 raft_consensus.cc:348] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.889405 7762 raft_consensus.cc:374] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:24:46.889426 7762 raft_consensus.cc:729] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a58d1c50a4be4a75a1909c505442a, State: Initialized, Role: FOLLOWER
I20260926 09:24:46.889566 7762 consensus_queue.cc:260] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.889833 7762 ts_tablet_manager.cc:1428] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:24:46.889896 7750 heartbeater.cc:503] Master 127.24.235.126:37637 was elected leader, sending a full tablet report...
I20260926 09:24:46.950161 7765 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:46.950281 7765 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.950627 7765 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439), 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489)
I20260926 09:24:46.953467 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.950899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:57349 (local address 127.24.235.66:33439)
0926 09:24:46.950950 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.950952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.950962 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:46.950993 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.950994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.951008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.951082 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.951248 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.951744 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.952662 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.953111 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.953191 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.953201 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.953357 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.953395 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:24:46.953498 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.950769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33439 (local address 127.24.235.67:57349)
0926 09:24:46.950825 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.950837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.950880 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.951088 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.951090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.951092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.951171 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.951174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.951828 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.951829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.952563 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.952564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.953188 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.953189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.953271 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.953274 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.953349 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.953392 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:24:46.953505 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.950912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:55971 (local address 127.24.235.65:46489)
0926 09:24:46.950955 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.950957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.951046 (+ 89us) server_negotiation.cc:407] Connection header received
0926 09:24:46.951122 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.951122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.951131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.951208 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:24:46.951365 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.951835 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.952736 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.953152 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.953242 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.953249 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.953400 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.953437 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:24:46.953608 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.950814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.24.235.67:55971)
0926 09:24:46.950974 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.950982 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:46.951056 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:46.951211 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:46.951211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:46.951213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:24:46.951289 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.951290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.951917 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.951918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.952648 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:46.952649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.953242 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.953243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:46.953316 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.953319 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:24:46.953392 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:24:46.953430 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260926 09:24:46.953912 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357c4cc7bc84ecc8e346037b1b2b21d" candidate_uuid: "793a58d1c50a4be4a75a1909c505442a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" is_pre_election: true
I20260926 09:24:46.954105 7572 raft_consensus.cc:2454] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a58d1c50a4be4a75a1909c505442a in term 0.
I20260926 09:24:46.954104 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357c4cc7bc84ecc8e346037b1b2b21d" candidate_uuid: "793a58d1c50a4be4a75a1909c505442a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3430798ff44e19ba3a567044e08d5" is_pre_election: true
I20260926 09:24:46.954239 7434 raft_consensus.cc:2454] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a58d1c50a4be4a75a1909c505442a in term 0.
I20260926 09:24:46.954372 7637 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a; no voters:
I20260926 09:24:46.954499 7765 raft_consensus.cc:2781] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:24:46.954555 7765 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:24:46.954610 7765 raft_consensus.cc:3037] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.955329 7765 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.955535 7765 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [CANDIDATE]: Term 1 election: Requested vote from peers 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439), 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489)
I20260926 09:24:46.955618 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357c4cc7bc84ecc8e346037b1b2b21d" candidate_uuid: "793a58d1c50a4be4a75a1909c505442a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
I20260926 09:24:46.955672 7434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357c4cc7bc84ecc8e346037b1b2b21d" candidate_uuid: "793a58d1c50a4be4a75a1909c505442a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3430798ff44e19ba3a567044e08d5"
I20260926 09:24:46.955698 7572 raft_consensus.cc:3037] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.955755 7434 raft_consensus.cc:3037] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:24:46.956434 7572 raft_consensus.cc:2454] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a58d1c50a4be4a75a1909c505442a in term 1.
I20260926 09:24:46.956516 7434 raft_consensus.cc:2454] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a58d1c50a4be4a75a1909c505442a in term 1.
I20260926 09:24:46.956600 7637 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a; no voters:
I20260926 09:24:46.956691 7765 raft_consensus.cc:2781] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:24:46.956840 7765 raft_consensus.cc:686] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 1 LEADER]: Becoming Leader. State: Replica: 793a58d1c50a4be4a75a1909c505442a, State: Running, Role: LEADER
I20260926 09:24:46.956954 7765 consensus_queue.cc:237] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:46.957800 7292 catalog_manager.cc:5495] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a58d1c50a4be4a75a1909c505442a (127.24.235.67). New cstate: current_term: 1 leader_uuid: "793a58d1c50a4be4a75a1909c505442a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } health_report { overall_health: UNKNOWN } } }
I20260926 09:24:46.971328 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.968891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.24.235.67:42935)
0926 09:24:46.968952 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:46.968954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:46.968964 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:24:46.969062 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:46.969063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:46.969088 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:46.969173 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:24:46.969320 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.969779 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.970467 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:46.970688 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:46.970767 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:46.970940 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:24:46.971033 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:24:46.971138 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:46.971195 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:24:46.971217 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:24:46.973152 7572 raft_consensus.cc:1261] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Refusing update from remote peer 793a58d1c50a4be4a75a1909c505442a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:46.973214 7434 raft_consensus.cc:1261] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Refusing update from remote peer 793a58d1c50a4be4a75a1909c505442a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:24:46.973520 7767 consensus_queue.cc:1035] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:46.973681 7765 consensus_queue.cc:1035] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [LEADER]: Connected to new peer: Peer: permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:24:46.975920 7770 mvcc.cc:205] Tried to move back new op lower bound from 7333538557839228928 to 7333538557776461824. Current Snapshot: MvccSnapshot[applied={T|T < 7333538557839228928}]
I20260926 09:24:46.976054 7771 mvcc.cc:205] Tried to move back new op lower bound from 7333538557839228928 to 7333538557776461824. Current Snapshot: MvccSnapshot[applied={T|T < 7333538557839228928}]
I20260926 09:24:46.978495 7291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:46.978603 7291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:24:46.979415 7291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Sending DeleteTablet for 3 replicas of tablet 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:46.979821 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:56560
I20260926 09:24:46.979943 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7330
I20260926 09:24:46.979954 7414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:45848
I20260926 09:24:46.980098 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357c4cc7bc84ecc8e346037b1b2b21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:24:46 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:44810
I20260926 09:24:46.980456 7780 tablet_replica.cc:330] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: stopping tablet replica
I20260926 09:24:46.980690 7780 raft_consensus.cc:2229] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:24:46.980865 7780 raft_consensus.cc:2258] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.980868 7781 tablet_replica.cc:330] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: stopping tablet replica
I20260926 09:24:46.980955 7781 raft_consensus.cc:2229] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:24:46.981115 7781 raft_consensus.cc:2258] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:24:46.981581 7781 ts_tablet_manager.cc:1905] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:46.981577 7780 ts_tablet_manager.cc:1905] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:24:46.983433 7781 ts_tablet_manager.cc:1918] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:46.983515 7781 log.cc:1196] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:46.983816 7781 ts_tablet_manager.cc:1939] T 9357c4cc7bc84ecc8e346037b1b2b21d P 793a58d1c50a4be4a75a1909c505442a: Deleting consensus metadata
I20260926 09:24:46.984062 7780 ts_tablet_manager.cc:1918] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:24:46.984205 7780 log.cc:1196] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:46.984617 7780 ts_tablet_manager.cc:1939] T 9357c4cc7bc84ecc8e346037b1b2b21d P 418f3551b43646b6b0bc9c4f44ac3f53: Deleting consensus metadata
I20260926 09:24:46.985176 7276 catalog_manager.cc:4841] TS 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): tablet 9357c4cc7bc84ecc8e346037b1b2b21d (table test-table [id=b114447cee4643d38d56f3eeac63e5c3]) successfully deleted
I20260926 09:24:46.985376 7278 catalog_manager.cc:4841] TS 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): tablet 9357c4cc7bc84ecc8e346037b1b2b21d (table test-table [id=b114447cee4643d38d56f3eeac63e5c3]) successfully deleted
W20260926 09:24:46.988229 7279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:46489 (error 108)
I20260926 09:24:46.988727 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7484
I20260926 09:24:46.988878 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:46.988694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:46489 (local address 127.0.0.1:45860)
0926 09:24:46.988771 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:46.988833 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 09:24:46.989478 7279 catalog_manager.cc:4584] TS 9ae3430798ff44e19ba3a567044e08d5 (127.24.235.65:46489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9357c4cc7bc84ecc8e346037b1b2b21d: Network error: Client connection negotiation failed: client connection to 127.24.235.65:46489: connect: Connection refused (error 111)
I20260926 09:24:46.995412 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7621
I20260926 09:24:47.002210 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7259
I20260926 09:24:47.014617 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--webserver_interface=127.24.235.126
--webserver_port=35479
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:47.097117 7783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:47.097357 7783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:47.097395 7783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:47.098582 7783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:24:47.098654 7783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:47.098681 7783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:47.098702 7783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:24:47.098738 7783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:24:47.100234 7783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:37637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=35479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:47.100517 7783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:47.100750 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:47.105633 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.105661 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.105664 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:47.106033 7783 server_base.cc:1034] running on GCE node
I20260926 09:24:47.106309 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:47.106596 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:47.107774 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790414687107769 us; error 37 us; skew 500 ppm
Sep 26 09:24:47 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:47.110427 7783 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:24:47.110888 7783 webserver.cc:460] Webserver started at http://127.24.235.126:35479/ using document root <none> and password file <none>
I20260926 09:24:47.111121 7783 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:47.111193 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:47.112536 7783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:47.113319 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.113517 7783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:47.113621 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "4f4314861bb847899b76b29f48154992"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "d6a84c46b8d7373284922719654fba4b"
server_key_iv: "8a3b3dd11c40d93867cc6f8527619825"
server_key_version: "encryptionkey@0"
I20260926 09:24:47.114010 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:47.124588 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:47.126899 7783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:47.127058 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:47.131050 7783 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:37637
I20260926 09:24:47.131083 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:37637 every 8 connection(s)
I20260926 09:24:47.131455 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:24:47.133307 7851 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:24:47.134003 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap starting.
I20260926 09:24:47.139619 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7783
I20260926 09:24:47.139909 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:46489
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=33765
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:47.143493 7851 log.cc:824] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:47.146255 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:24:47.146661 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992: Bootstrap complete.
I20260926 09:24:47.149329 7851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:47.149580 7851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4314861bb847899b76b29f48154992, State: Initialized, Role: FOLLOWER
I20260926 09:24:47.149763 7851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:47.149850 7851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:24:47.149912 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:24:47.149972 7851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:24:47.151054 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:47.151182 7851 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4314861bb847899b76b29f48154992; no voters:
I20260926 09:24:47.151404 7851 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:24:47.151445 7855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:24:47.151568 7855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [term 3 LEADER]: Becoming Leader. State: Replica: 4f4314861bb847899b76b29f48154992, State: Running, Role: LEADER
I20260926 09:24:47.151664 7855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } }
I20260926 09:24:47.151710 7851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:24:47.151939 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4314861bb847899b76b29f48154992. Latest consensus state: current_term: 3 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:47.152164 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:47.152098 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4f4314861bb847899b76b29f48154992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4314861bb847899b76b29f48154992" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 37637 } } }
I20260926 09:24:47.152413 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4314861bb847899b76b29f48154992 [sys.catalog]: This master's current role is: LEADER
I20260926 09:24:47.152555 7863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:24:47.153373 7863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:24:47.153474 7863 catalog_manager.cc:1259] Loaded cluster ID: 55fce923d21a45b69f4bdf846f8bfd2a
I20260926 09:24:47.153506 7863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 09:24:47.153829 7863 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcce717b9ad google::LogMessage::Fail() at ??:0
@ 0x7fcce717e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcce717b4de google::LogMessage::Flush() at ??:0
@ 0x7fcce717ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fccef09c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fccef0a58b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcce90490d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcce903e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcce9ea26db start_thread at ??:0
@ 0x7fcce5c7961f clone at ??:0
W20260926 09:24:47.246236 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:47.246441 7853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:47.246465 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:47.247718 7853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:47.247766 7853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:47.247792 7853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:24:47.249356 7853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:46489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=33765
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:47.249596 7853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:47.249842 7853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:47.255944 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.255987 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:47.256289 7853 server_base.cc:1034] running on GCE node
W20260926 09:24:47.255932 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:47.256527 7853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:47.256809 7853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:47.257965 7853 hybrid_clock.cc:648] HybridClock initialized: now 1790414687257943 us; error 36 us; skew 500 ppm
Sep 26 09:24:47 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:47.261520 7853 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:24:47.262084 7853 webserver.cc:460] Webserver started at http://127.24.235.65:33765/ using document root <none> and password file <none>
I20260926 09:24:47.262334 7853 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:47.262403 7853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:47.264621 7853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:24:47.265801 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.265964 7853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:24:47.266065 7853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "9ae3430798ff44e19ba3a567044e08d5"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "a74b712a764ff61d057458845168fc18"
server_key_iv: "ce65e01b4b6414ba6a9af483ccde947e"
server_key_version: "encryptionkey@0"
I20260926 09:24:47.266436 7853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:47.286586 7853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:47.289001 7853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:47.289227 7853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:47.289496 7853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:47.289964 7904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:24:47.291966 7853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:24:47.292053 7853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:24:47.292120 7853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:24:47.292817 7853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:24:47.292874 7853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:24:47.292945 7904 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap starting.
I20260926 09:24:47.300374 7853 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:46489
I20260926 09:24:47.300776 7853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:24:47.304471 8011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:46489 every 8 connection(s)
I20260926 09:24:47.304934 7904 log.cc:824] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:24:47.305719 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 7853
I20260926 09:24:47.306106 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:33439
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=43961
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:24:47.307817 7904 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:24:47.308141 7904 tablet_bootstrap.cc:492] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Bootstrap complete.
I20260926 09:24:47.309562 7904 ts_tablet_manager.cc:1397] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.014s
I20260926 09:24:47.311198 7904 raft_consensus.cc:348] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:47.311569 7904 raft_consensus.cc:729] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3430798ff44e19ba3a567044e08d5, State: Initialized, Role: FOLLOWER
I20260926 09:24:47.311758 7904 consensus_queue.cc:260] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:47.312058 7904 ts_tablet_manager.cc:1428] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260926 09:24:47.392098 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:47.392362 8014 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:24:47.301579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:45619)
0926 09:24:47.301763 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.301808 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:47.302309 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:47.392154 (+ 89845us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
W20260926 09:24:47.392378 8016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:47.392416 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:47.392759 8012 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:37637 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:37637: Client connection negotiation failed: client connection to 127.24.235.126:37637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:24:47.393221 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.393061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:52417)
0926 09:24:47.393148 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.393193 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 09:24:47.393652 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.393544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:33801)
0926 09:24:47.393600 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.393630 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:24:47.393807 8012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:24:47.394143 8016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:47.394186 8016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:47.394244 8016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:24:47.395777 8016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=43961
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:47.396032 8016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:47.396282 8016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:47.401394 8023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.401394 8025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.401446 8022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:47.401664 8016 server_base.cc:1034] running on GCE node
I20260926 09:24:47.401896 8016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:47.402103 8016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:47.403285 8016 hybrid_clock.cc:648] HybridClock initialized: now 1790414687403271 us; error 30 us; skew 500 ppm
Sep 26 09:24:47 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:47.406754 8016 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:24:47.407375 8016 webserver.cc:460] Webserver started at http://127.24.235.66:43961/ using document root <none> and password file <none>
I20260926 09:24:47.407657 8016 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:47.407735 8016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:47.409770 8016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:24:47.410544 8032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.410764 8016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:24:47.410861 8016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "418f3551b43646b6b0bc9c4f44ac3f53"
format_stamp: "Formatted at 2026-09-26 09:24:45 on dist-test-slave-7zlw"
server_key: "0b17396fb9d3da94dc15092b68e8a229"
server_key_iv: "f70e476e1f6ae12d27437726a99a8088"
server_key_version: "encryptionkey@0"
I20260926 09:24:47.411366 8016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:47.425216 8016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:47.427184 8016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:47.427331 8016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:47.427577 8016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:47.427891 8016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:47.427946 8016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.427989 8016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:47.428017 8016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.434178 8016 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33439
I20260926 09:24:47.434252 8145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33439 every 8 connection(s)
I20260926 09:24:47.434616 8016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:24:47.435778 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.435286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:33325)
0926 09:24:47.435453 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.435599 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
W20260926 09:24:47.436215 8146 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:37637 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:37637: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
I20260926 09:24:47.436502 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.436412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:38905)
0926 09:24:47.436459 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.436484 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:24:47.436830 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.436737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:57229)
0926 09:24:47.436786 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.436816 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:24:47.436946 8146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:24:47.442479 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8016
I20260926 09:24:47.442696 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:42935
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:37637
--webserver_port=35175
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:24:47.526351 8149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:24:47.526612 8149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:24:47.526654 8149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:24:47.527804 8149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:24:47.527872 8149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:24:47.527918 8149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:24:47.529538 8149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:42935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=35175
--tserver_master_addrs=127.24.235.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:24:47.529810 8149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:24:47.530051 8149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:24:47.535399 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.535410 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:24:47.535670 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:24:47.535650 8149 server_base.cc:1034] running on GCE node
I20260926 09:24:47.535916 8149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:24:47.536175 8149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:24:47.537321 8149 hybrid_clock.cc:648] HybridClock initialized: now 1790414687537294 us; error 42 us; skew 500 ppm
Sep 26 09:24:47 dist-test-slave-7zlw krb5kdc[6712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:24:47.539831 8149 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:24:47.540295 8149 webserver.cc:460] Webserver started at http://127.24.235.67:35175/ using document root <none> and password file <none>
I20260926 09:24:47.540477 8149 fs_manager.cc:359] Metadata directory not provided
I20260926 09:24:47.540517 8149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:24:47.541918 8149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:24:47.542502 8164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.542707 8149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 09:24:47.542783 8149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "793a58d1c50a4be4a75a1909c505442a"
format_stamp: "Formatted at 2026-09-26 09:24:46 on dist-test-slave-7zlw"
server_key: "25ba1a56f26f9f201830e769dc66ecc1"
server_key_iv: "a197d75078220fde125a743593ee9756"
server_key_version: "encryptionkey@0"
I20260926 09:24:47.543099 8149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:24:47.561182 8149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:24:47.563324 8149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:24:47.563432 8149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:24:47.563612 8149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:24:47.563892 8149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:24:47.563923 8149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.563946 8149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:24:47.563958 8149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:24:47.570305 8149 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:42935
I20260926 09:24:47.570381 8277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:42935 every 8 connection(s)
I20260926 09:24:47.570741 8149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:24:47.571890 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.571447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:42151)
0926 09:24:47.571607 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.571750 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260926 09:24:47.572273 8278 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:37637 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:37637: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
I20260926 09:24:47.572580 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.572469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:33687)
0926 09:24:47.572505 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.572552 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 09:24:47.572944 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:47.572847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:43597)
0926 09:24:47.572892 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:47.572916 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:24:47.573066 8278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:24:47.578264 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8149
I20260926 09:24:48.394596 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.394296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:60459)
0926 09:24:48.394450 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:48.394511 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260926 09:24:48.437484 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.437282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:45123)
0926 09:24:48.437405 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:48.437437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 09:24:48.573793 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.573474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:32929)
0926 09:24:48.573691 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:48.573736 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260926 09:24:48.847018 8285 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:48.847167 8285 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:48.847638 8285 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935), 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
Sep 26 09:24:48 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.66@KRBTEST.COM
Sep 26 09:24:48 dist-test-slave-7zlw krb5kdc[6712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414687, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.67@KRBTEST.COM
I20260926 09:24:48.860931 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.847885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:42935 (local address 127.24.235.65:45845)
0926 09:24:48.848020 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:48.848034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:48.848102 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:48.849078 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:48.849082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:48.849628 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:48.849933 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:48.849941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.851701 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.851702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:48.853141 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:48.853144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.853989 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:48.854137 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:48.854803 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:48.854815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:48.857249 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:48.859256 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:48.859266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:48.859547 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:48.859846 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:48.859847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:48.859963 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:48.860368 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:48.860375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:48.860505 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 09:24:48.860605 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 09:24:48.860814 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":56,"thread_start_us":59,"threads_started":1}
I20260926 09:24:48.861054 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.847994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:36021 (local address 127.24.235.66:33439)
0926 09:24:48.848173 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:48.848180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:48.848218 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:24:48.848762 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:48.848765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:48.848825 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:48.848982 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:48.850086 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.851544 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853249 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.853877 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853992 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:48.856999 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:48.857004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:48.857004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:48.857017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:48.859163 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859663 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:48.859667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:48.859667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:48.859766 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:48.860057 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:48.860058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:48.860058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:48.860302 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:48.860396 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:48.860757 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 09:24:48.860904 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260926 09:24:48.861224 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.847853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33439 (local address 127.24.235.65:36021)
0926 09:24:48.847917 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:48.847929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:24:48.847989 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:24:48.849078 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:24:48.849082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:24:48.849628 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:24:48.849933 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:48.849941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.851709 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.851710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:48.853141 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:24:48.853144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.853987 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:24:48.854137 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:48.854824 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:24:48.854829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:24:48.856859 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:24:48.859285 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:48.859287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:48.859563 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:48.859853 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:24:48.859854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:24:48.859976 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:24:48.860391 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:24:48.860392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:24:48.860509 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:24:48.860605 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 09:24:48.860906 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":123}
I20260926 09:24:48.861413 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:48.847955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:45845 (local address 127.24.235.67:42935)
0926 09:24:48.848102 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 09:24:48.848113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:24:48.848150 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:24:48.848799 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:24:48.848802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:24:48.848841 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:24:48.848983 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:24:48.850086 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.851543 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853254 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:24:48.853877 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:24:48.853992 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:24:48.857365 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:24:48.857369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:24:48.857369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:24:48.857379 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:24:48.859149 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:48.859682 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:48.859685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:48.859685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:48.859769 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:24:48.860037 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:24:48.860038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:24:48.860038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:24:48.860283 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:24:48.860373 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:24:48.861178 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 09:24:48.861311 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
W20260926 09:24:48.862293 7898 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
W20260926 09:24:48.862399 7900 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:48.862447 7900 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3430798ff44e19ba3a567044e08d5; no voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a
I20260926 09:24:48.862558 8285 raft_consensus.cc:2726] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:24:49.395406 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:49.395146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:35819)
0926 09:24:49.395303 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:49.395354 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260926 09:24:49.438175 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:49.437945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:35693)
0926 09:24:49.438074 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:49.438112 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260926 09:24:49.574529 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:49.574308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:57861)
0926 09:24:49.574445 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:49.574486 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260926 09:24:50.396198 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:50.395957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:33303)
0926 09:24:50.396107 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:50.396149 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 09:24:50.438999 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:50.438756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:34427)
0926 09:24:50.438911 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:50.438955 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260926 09:24:50.575469 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:50.575248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:46237)
0926 09:24:50.575382 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:50.575421 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260926 09:24:51.045753 8295 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:51.045897 8295 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:51.046088 8295 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935), 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
W20260926 09:24:51.046512 7900 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
W20260926 09:24:51.046617 7898 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:51.046692 7898 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3430798ff44e19ba3a567044e08d5; no voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a
I20260926 09:24:51.046816 8295 raft_consensus.cc:2726] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:24:51.396972 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:51.396741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:59531)
0926 09:24:51.396883 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:51.396923 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260926 09:24:51.439795 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:51.439536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:46781)
0926 09:24:51.439692 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:51.439734 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
I20260926 09:24:51.576267 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:51.576020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:48365)
0926 09:24:51.576174 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:51.576218 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260926 09:24:52.397949 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:52.397695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:54435)
0926 09:24:52.397848 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:52.397901 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260926 09:24:52.440567 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:52.440327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:60533)
0926 09:24:52.440474 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:52.440518 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260926 09:24:52.577116 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:52.576827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:42535)
0926 09:24:52.576975 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:52.577039 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260926 09:24:52.933780 8302 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:52.933915 8302 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:52.934214 8302 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935), 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
W20260926 09:24:52.934661 7898 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
W20260926 09:24:52.934747 7900 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:52.934829 7900 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3430798ff44e19ba3a567044e08d5; no voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a
I20260926 09:24:52.934940 8302 raft_consensus.cc:2726] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:24:53.398684 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:53.398438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:51707)
0926 09:24:53.398596 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:53.398638 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260926 09:24:53.441455 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:53.441192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:51527)
0926 09:24:53.441368 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:53.441406 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260926 09:24:53.578083 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:53.577676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:51325)
0926 09:24:53.577852 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:53.577896 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260926 09:24:54.399555 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:54.399270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:34973)
0926 09:24:54.399441 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:54.399500 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260926 09:24:54.442270 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:54.442044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:44407)
0926 09:24:54.442185 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:54.442225 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260926 09:24:54.578789 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:54.578557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:58075)
0926 09:24:54.578698 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:54.578742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260926 09:24:55.391574 8309 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:24:55.391700 8309 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:24:55.392020 8309 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935), 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
W20260926 09:24:55.392438 7898 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
W20260926 09:24:55.392508 7900 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:24:55.392586 7900 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3430798ff44e19ba3a567044e08d5; no voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a
I20260926 09:24:55.392724 8309 raft_consensus.cc:2726] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:24:55.400374 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:55.400124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:34697)
0926 09:24:55.400284 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:55.400329 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260926 09:24:55.442998 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:55.442797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:56025)
0926 09:24:55.442920 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:55.442951 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260926 09:24:55.579540 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:55.579316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:33081)
0926 09:24:55.579449 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:55.579493 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260926 09:24:56.401113 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:56.400847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:48625)
0926 09:24:56.400997 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:56.401063 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260926 09:24:56.443825 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:56.443570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:41243)
0926 09:24:56.443728 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:56.443774 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260926 09:24:56.580413 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:56.580161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:38195)
0926 09:24:56.580313 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:56.580364 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260926 09:24:57.401955 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:57.401732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:35325)
0926 09:24:57.401874 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:57.401907 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260926 09:24:57.444584 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:57.444377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:47781)
0926 09:24:57.444504 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:57.444537 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260926 09:24:57.581362 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:57.581130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:42939)
0926 09:24:57.581280 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:57.581316 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260926 09:24:58.402812 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:58.402523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:44533)
0926 09:24:58.402721 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:58.402765 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260926 09:24:58.445431 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:58.445152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:50961)
0926 09:24:58.445333 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:58.445382 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260926 09:24:58.582183 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:58.581904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:42259)
0926 09:24:58.582082 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:58.582128 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
I20260926 09:24:59.403563 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:59.403323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:39495)
0926 09:24:59.403468 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:59.403512 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 09:24:59.446326 8323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:59.446040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:58389)
0926 09:24:59.446201 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:59.446253 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260926 09:24:59.582923 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:24:59.582705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:39185)
0926 09:24:59.582837 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 09:24:59.582877 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 09:25:00.404609 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:00.404324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:44123)
0926 09:25:00.404509 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:00.404558 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260926 09:25:00.447132 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:00.446890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:54255)
0926 09:25:00.447030 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:00.447085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260926 09:25:00.583798 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:00.583528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:52913)
0926 09:25:00.583704 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:00.583750 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260926 09:25:01.405480 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:01.405191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:43489)
0926 09:25:01.405368 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:01.405428 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260926 09:25:01.447925 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:01.447631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:49573)
0926 09:25:01.447833 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:01.447876 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":62,"threads_started":1}
I20260926 09:25:01.584585 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:01.584344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:53073)
0926 09:25:01.584494 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:01.584537 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260926 09:25:02.327292 8331 raft_consensus.cc:482] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:02.327389 8331 raft_consensus.cc:504] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a58d1c50a4be4a75a1909c505442a" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 42935 } } peers { permanent_uuid: "418f3551b43646b6b0bc9c4f44ac3f53" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33439 } } peers { permanent_uuid: "9ae3430798ff44e19ba3a567044e08d5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 46489 } }
I20260926 09:25:02.327674 8331 leader_election.cc:288] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935), 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439)
W20260926 09:25:02.328049 7900 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 793a58d1c50a4be4a75a1909c505442a (127.24.235.67:42935): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
W20260926 09:25:02.328123 7898 leader_election.cc:341] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 418f3551b43646b6b0bc9c4f44ac3f53 (127.24.235.66:33439): Not found: Tablet not found: 9357c4cc7bc84ecc8e346037b1b2b21d
I20260926 09:25:02.328225 7898 leader_election.cc:302] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3430798ff44e19ba3a567044e08d5; no voters: 418f3551b43646b6b0bc9c4f44ac3f53, 793a58d1c50a4be4a75a1909c505442a
I20260926 09:25:02.328357 8331 raft_consensus.cc:2726] T 9357c4cc7bc84ecc8e346037b1b2b21d P 9ae3430798ff44e19ba3a567044e08d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:25:02.406358 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:02.406118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.65:58855)
0926 09:25:02.406267 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:02.406310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260926 09:25:02.448832 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:02.448580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.66:52639)
0926 09:25:02.448739 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:02.448785 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260926 09:25:02.585669 8334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:02.585386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:37637 (local address 127.24.235.67:60047)
0926 09:25:02.585563 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:02.585618 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:37637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260926 09:25:02.587463 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 7853
I20260926 09:25:02.593674 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8016
I20260926 09:25:02.598598 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8149
2026-09-26T09:25:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18375 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:25:02 dist-test-slave-7zlw krb5kdc[8337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:25:02 dist-test-slave-7zlw krb5kdc[8337](info): set up 2 sockets
Sep 26 09:25:02 dist-test-slave-7zlw krb5kdc[8337](info): commencing operation
krb5kdc: starting...
W20260926 09:25:03.701332 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:25:03 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:25:03.829006 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:03.911597 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:03.911880 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:03.911917 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:03.913146 8350 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:25:03.913228 8350 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:25:03.913249 8350 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:25:03.913268 8350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:25:03.913295 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:03.913326 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:03.913349 8350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:25:03.913370 8350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:25:03.914919 8350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:03.915234 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:03.915481 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:03.920826 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:03.920835 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:03.920877 8350 server_base.cc:1034] running on GCE node
W20260926 09:25:03.920835 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:03.921377 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:03.921728 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:03.922868 8350 hybrid_clock.cc:648] HybridClock initialized: now 1790414703922849 us; error 38 us; skew 500 ppm
Sep 26 09:25:03 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414703, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:03.925518 8350 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:25:03.925995 8350 webserver.cc:460] Webserver started at http://127.24.235.126:42923/ using document root <none> and password file <none>
I20260926 09:25:03.926244 8350 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:03.926291 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:03.926429 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:25:03.927531 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "7cb09fd09a0d4ed88caa0d798b7525f1"
format_stamp: "Formatted at 2026-09-26 09:25:03 on dist-test-slave-7zlw"
server_key: "a78577027cce75f492aba3734a606384"
server_key_iv: "3b05d95026c45a6a7b4a1c23605158e0"
server_key_version: "encryptionkey@0"
I20260926 09:25:03.927911 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "7cb09fd09a0d4ed88caa0d798b7525f1"
format_stamp: "Formatted at 2026-09-26 09:25:03 on dist-test-slave-7zlw"
server_key: "a78577027cce75f492aba3734a606384"
server_key_iv: "3b05d95026c45a6a7b4a1c23605158e0"
server_key_version: "encryptionkey@0"
I20260926 09:25:03.929375 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:25:03.930225 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:03.930529 8350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:03.930636 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "7cb09fd09a0d4ed88caa0d798b7525f1"
format_stamp: "Formatted at 2026-09-26 09:25:03 on dist-test-slave-7zlw"
server_key: "a78577027cce75f492aba3734a606384"
server_key_iv: "3b05d95026c45a6a7b4a1c23605158e0"
server_key_version: "encryptionkey@0"
I20260926 09:25:03.930755 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:03.960640 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:03.963224 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:03.963411 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:03.967792 8350 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41015
I20260926 09:25:03.967834 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41015 every 8 connection(s)
I20260926 09:25:03.968202 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:25:03.969174 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:03.971501 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap starting.
I20260926 09:25:03.972148 8418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:03.972468 8418 log.cc:824] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:03.973238 8418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: No bootstrap required, opened a new log
I20260926 09:25:03.974967 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8350
I20260926 09:25:03.974903 8418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:03.975052 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:25:03.975065 8418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:03.975117 8418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Initialized, Role: FOLLOWER
I20260926 09:25:03.975271 8418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:03.975284 25517 external_mini_cluster.cc:1428] Setting key 8daf5d2856e45fdeb8818959604a49ae
I20260926 09:25:03.975359 8418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:25:03.975402 8418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:25:03.975456 8418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:03.976738 8418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:03.976832 8418 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb09fd09a0d4ed88caa0d798b7525f1; no voters:
I20260926 09:25:03.977052 8418 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:25:03.977069 8424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:25:03.977192 8424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Running, Role: LEADER
I20260926 09:25:03.977334 8418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:25:03.977348 8424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:03.977960 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:03.978055 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:03.978214 8426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb09fd09a0d4ed88caa0d798b7525f1. Latest consensus state: current_term: 1 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:03.978260 8426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:03.978566 8433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:25:03.979095 8433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:25:03.981299 8433 catalog_manager.cc:1347] Generated new cluster ID: dd5c411e29ea4087905f8ee9e6ba8a9c
I20260926 09:25:03.981359 8433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:25:03 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:03.986572 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:03.975852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.24.235.126:41015)
0926 09:25:03.976069 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:03.976080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:03.976119 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:25:03.976280 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:03.976290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:03.976361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:03.976529 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:03.977391 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:03.978810 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:03.979946 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:03.980475 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:03.980600 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:03.983137 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:03.983141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:03.983141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:03.983154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:03.984931 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:03.985342 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:03.985346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:03.985346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:03.985459 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:03.985680 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:03.985681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:03.985682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:03.985943 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:03.986102 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:03.986218 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 09:25:03.986372 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260926 09:25:03.987847 8433 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:25:03.988600 8433 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:25:04.002475 8433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Generated new TSK 0
I20260926 09:25:04.002754 8433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:25:04.040125 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:04.161280 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:04.161571 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:04.161607 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:04.163380 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:04.163436 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:04.163475 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:25:04.165797 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:04.166047 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:04.166299 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:04.171830 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:04.171837 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:04.171837 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:04.171893 8445 server_base.cc:1034] running on GCE node
I20260926 09:25:04.172307 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:04.172521 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:04.173663 8445 hybrid_clock.cc:648] HybridClock initialized: now 1790414704173643 us; error 37 us; skew 500 ppm
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:04.176326 8445 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:25:04.176756 8445 webserver.cc:460] Webserver started at http://127.24.235.65:40463/ using document root <none> and password file <none>
I20260926 09:25:04.176959 8445 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:04.176999 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:04.177177 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:25:04.178371 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "f0f0b4775ea412d769943caf5d3e3f7b"
server_key_iv: "c2ecda30c216e1085965a2a222f00f3e"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.178697 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "f0f0b4775ea412d769943caf5d3e3f7b"
server_key_iv: "c2ecda30c216e1085965a2a222f00f3e"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.180019 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:25:04.180819 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.181057 8445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:04.181160 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "f0f0b4775ea412d769943caf5d3e3f7b"
server_key_iv: "c2ecda30c216e1085965a2a222f00f3e"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.181250 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:04.190944 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:04.193337 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:04.193503 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:04.193745 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:04.194063 8445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:04.194118 8445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.194161 8445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:04.194190 8445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.200342 8445 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44609
I20260926 09:25:04.200418 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44609 every 8 connection(s)
I20260926 09:25:04.200790 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:25:04.206871 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8445
I20260926 09:25:04.207000 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:25:04.207234 25517 external_mini_cluster.cc:1428] Setting key dada9e5d748e38fd43be168577141551
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:04.211193 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.201860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:33259 (local address 127.24.235.126:41015)
0926 09:25:04.201965 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.201967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.202039 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:25:04.202592 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.202593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.202610 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.202685 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:04.203783 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.204303 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.205403 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.205602 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.205659 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.208025 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:04.208027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:04.208027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:04.208045 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:04.209555 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.210031 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.210033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.210033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.210118 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.210466 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.210467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.210467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.210610 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:04.210678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.211028 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.211123 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 09:25:04.211457 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.201700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:33259)
0926 09:25:04.201885 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.201932 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.202443 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.202794 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.202799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.203424 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:04.203644 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.203648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.204389 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.204389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.205326 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.205328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.205681 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.205682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.205783 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.206271 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:04.206279 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:04.207909 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:04.209665 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.209674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.209677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.209920 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.210206 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.210207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.210207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.210359 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.210691 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:04.210697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:04.210883 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.211005 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.211206 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260926 09:25:04.212009 8574 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:04.212124 8574 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:04.212311 8574 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:04.212983 8382 ts_manager.cc:194] Registered new tserver with Master: d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609)
I20260926 09:25:04.214032 8382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:33259
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:25:04.236788 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:04.324357 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:04.324594 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:04.324620 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:04.325870 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:04.325917 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:04.325945 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:25:04.327765 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:04.328059 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:04.328334 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:04.333552 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:04.333663 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:04.333808 8579 server_base.cc:1034] running on GCE node
W20260926 09:25:04.333839 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:04.334091 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:04.334339 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:04.335501 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790414704335476 us; error 39 us; skew 500 ppm
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:04.338286 8579 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:25:04.338763 8579 webserver.cc:460] Webserver started at http://127.24.235.66:46801/ using document root <none> and password file <none>
I20260926 09:25:04.338990 8579 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:04.339058 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:04.339215 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:25:04.340351 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "365d262b78c01f4139340761fd1da777"
server_key_iv: "f5e9babee6d4b01bd8066d94969c938a"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.340713 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "365d262b78c01f4139340761fd1da777"
server_key_iv: "f5e9babee6d4b01bd8066d94969c938a"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.342262 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:25:04.343047 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.343217 8579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.343317 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "365d262b78c01f4139340761fd1da777"
server_key_iv: "f5e9babee6d4b01bd8066d94969c938a"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.343405 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:04.362542 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:04.364526 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:04.364701 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:04.364955 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:04.365301 8579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:04.365355 8579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.365396 8579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:04.365437 8579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.371894 8579 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:36835
I20260926 09:25:04.371958 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:36835 every 8 connection(s)
I20260926 09:25:04.372319 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:25:04.372737 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8579
I20260926 09:25:04.372841 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:25:04.373108 25517 external_mini_cluster.cc:1428] Setting key 1c770c0152ea356b131e2d4bd7378d5d
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
I20260926 09:25:04.382753 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.373446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:47549 (local address 127.24.235.126:41015)
0926 09:25:04.373512 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.373514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.373542 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:25:04.374162 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.374164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.374182 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.374226 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:04.375198 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.375717 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.376742 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.377123 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.377222 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.379647 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:04.379649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:04.379649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:04.379657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:04.381346 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.381727 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.381729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.381729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.381796 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.382041 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.382043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.382043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.382210 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:04.382286 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.382525 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.382627 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:25:04.382871 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.373232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:47549)
0926 09:25:04.373402 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.373456 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.374061 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.374294 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.374298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.374890 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:04.375095 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.375100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.375816 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.375817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.376630 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.376632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.377265 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.377266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.377396 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.378041 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:04.378049 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:04.379554 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:04.381423 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.381430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.381432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.381650 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.381876 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.381877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.381877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.381976 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.382281 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:04.382285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:04.382400 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.382481 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.382607 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260926 09:25:04.383471 8708 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:04.383585 8708 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:04.383859 8708 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:04.384299 8382 ts_manager.cc:194] Registered new tserver with Master: cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
I20260926 09:25:04.384685 8382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:47549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:25:04.404812 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:04.486799 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:04.487015 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:04.487038 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:04.488284 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:04.488330 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:04.488376 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:25:04.489910 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:04.490156 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:04.490391 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:04.495528 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:04.495558 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:04.495528 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:04.495833 8713 server_base.cc:1034] running on GCE node
I20260926 09:25:04.496068 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:04.496315 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:04.497468 8713 hybrid_clock.cc:648] HybridClock initialized: now 1790414704497456 us; error 32 us; skew 500 ppm
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:04.500291 8713 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:25:04.500739 8713 webserver.cc:460] Webserver started at http://127.24.235.67:38797/ using document root <none> and password file <none>
I20260926 09:25:04.500969 8713 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:04.501056 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:04.501214 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:25:04.502288 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "e5b064c4efadbfdd7574f516402224ec"
server_key_iv: "e61b0361dfab786c81d057d29273bf74"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.502729 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "e5b064c4efadbfdd7574f516402224ec"
server_key_iv: "e61b0361dfab786c81d057d29273bf74"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.504130 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:25:04.504877 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.505090 8713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:04.505190 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "e5b064c4efadbfdd7574f516402224ec"
server_key_iv: "e61b0361dfab786c81d057d29273bf74"
server_key_version: "encryptionkey@0"
I20260926 09:25:04.505275 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:04.529603 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:04.531917 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:04.532086 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:04.532334 8713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:04.532682 8713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:04.532750 8713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.532793 8713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:04.532820 8713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:04.539006 8713 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:34459
I20260926 09:25:04.539065 8841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:34459 every 8 connection(s)
I20260926 09:25:04.539412 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:25:04.540138 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8713
I20260926 09:25:04.540248 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:25:04.540473 25517 external_mini_cluster.cc:1428] Setting key cf9a4eeec58795f75f5edf3c6a080ec6
Sep 26 09:25:04 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414704, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:04.548867 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.540185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:39265 (local address 127.24.235.126:41015)
0926 09:25:04.540234 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.540236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.540590 (+ 354us) server_negotiation.cc:407] Connection header received
0926 09:25:04.541278 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.541279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.541291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.541351 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:04.542327 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.542817 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.543791 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.543973 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.544028 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.546289 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:04.546290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:04.546290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:04.546297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:04.547545 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.547950 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.547951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.547951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.548019 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.548270 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.548271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.548271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.548400 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:04.548471 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.548723 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.548797 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:25:04.548985 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.540187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:39265)
0926 09:25:04.540387 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.540441 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.541139 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.541443 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.541448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.542042 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:04.542251 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.542255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.542893 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.542894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.543710 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.543712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.544045 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.544046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.544147 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.544706 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:04.544716 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:04.546188 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:04.547623 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.547629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.547631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.547856 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.548089 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:04.548089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:04.548090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:04.548184 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:04.548464 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:04.548468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:04.548581 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.548665 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.548785 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260926 09:25:04.549512 8842 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:04.549616 8842 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:04.549826 8842 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:04.551095 8382 ts_manager.cc:194] Registered new tserver with Master: 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459)
I20260926 09:25:04.551466 8382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:39265
I20260926 09:25:04.552270 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:25:04.559378 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.553757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.24.235.126:41015)
0926 09:25:04.553812 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.553813 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.553901 (+ 88us) server_negotiation.cc:407] Connection header received
0926 09:25:04.553967 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.553967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.553976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.554023 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:04.554560 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.555027 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.555764 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.555938 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.555985 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.557098 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:04.557099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:04.557099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:04.557105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:04.558425 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.558736 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.558737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.558737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.558788 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:04.558940 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:04.558940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:04.558940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:04.559071 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:04.559146 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.559257 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.559316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 09:25:04.560884 8382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:25:04.562527 8382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:25:04.570145 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.24.235.65:44609)
0926 09:25:04.565747 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.565751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.565762 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:25:04.565880 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.565884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.565900 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.565955 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:04.566197 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.567269 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.568993 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.569681 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569777 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.569842 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.569989 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.570037 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":87,"threads_started":1}
I20260926 09:25:04.570590 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.24.235.66:36835)
0926 09:25:04.565524 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.565528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.565932 (+ 404us) server_negotiation.cc:407] Connection header received
0926 09:25:04.565979 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.565981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.565996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.566074 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:04.567403 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.568325 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569421 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.570146 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.570209 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.570277 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.570480 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.570529 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
I20260926 09:25:04.570614 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.24.235.67:34459)
0926 09:25:04.565546 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.565551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.565564 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:25:04.565614 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.565616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.565634 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.565807 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:04.566137 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.567284 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569087 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.569718 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569790 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.569841 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.570502 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.570548 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":105,"threads_started":1}
I20260926 09:25:04.570747 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34459 (local address 127.0.0.1:60390)
0926 09:25:04.565278 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.565321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.565389 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.565889 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.565892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.565895 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:04.565982 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.565983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.567397 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.567397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.568897 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.568898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.569876 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.569980 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.569983 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.570037 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.570105 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":77,"thread_start_us":49,"threads_started":1}
I20260926 09:25:04.570778 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:36835 (local address 127.0.0.1:48118)
0926 09:25:04.565805 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.565822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.565874 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.566068 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.566069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.566073 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:04.566134 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.566137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.568436 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.568438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.569332 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.569335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.570318 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.570320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.570385 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.570389 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.570444 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.570486 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260926 09:25:04.570760 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.565108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.0.0.1:45288)
0926 09:25:04.565712 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.565723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.565768 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.566068 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.566069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.566071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:04.566134 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.566136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.567395 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.567396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.568895 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.568898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.569781 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.569782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.569865 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.569881 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.569970 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.570339 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":102,"thread_start_us":89,"threads_started":1}
I20260926 09:25:04.571655 8642 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (DEFAULT_TABLE table=test-table [id=7791410430d0430cb9767d4fc6071023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:04.571852 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (DEFAULT_TABLE table=test-table [id=7791410430d0430cb9767d4fc6071023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:04.572053 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:04.572190 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:04.573398 8776 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (DEFAULT_TABLE table=test-table [id=7791410430d0430cb9767d4fc6071023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:04.573714 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:04.575582 8860 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap starting.
I20260926 09:25:04.576321 8860 tablet_bootstrap.cc:654] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:04.576648 8861 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Bootstrap starting.
I20260926 09:25:04.576812 8859 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Bootstrap starting.
I20260926 09:25:04.577054 8860 log.cc:824] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:04.577425 8861 tablet_bootstrap.cc:654] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:04.577538 8859 tablet_bootstrap.cc:654] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:04.577971 8861 log.cc:824] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:04.578012 8860 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: No bootstrap required, opened a new log
I20260926 09:25:04.578073 8860 ts_tablet_manager.cc:1397] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:25:04.578146 8859 log.cc:824] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:04.578671 8861 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: No bootstrap required, opened a new log
I20260926 09:25:04.578747 8861 ts_tablet_manager.cc:1397] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:25:04.578943 8859 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: No bootstrap required, opened a new log
I20260926 09:25:04.579012 8859 ts_tablet_manager.cc:1397] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:25:04.580029 8860 raft_consensus.cc:348] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.580196 8860 raft_consensus.cc:374] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:04.580240 8860 raft_consensus.cc:729] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c110a2424844b7a50e3a9e6531b9cd, State: Initialized, Role: FOLLOWER
I20260926 09:25:04.580377 8860 consensus_queue.cc:260] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.580682 8574 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
I20260926 09:25:04.580727 8861 raft_consensus.cc:348] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.580919 8861 raft_consensus.cc:374] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:04.580668 8860 ts_tablet_manager.cc:1428] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:25:04.580993 8861 raft_consensus.cc:729] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a409cd42ddc4ae8a7e9f68ad55b78db, State: Initialized, Role: FOLLOWER
I20260926 09:25:04.581072 8859 raft_consensus.cc:348] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.581146 8861 consensus_queue.cc:260] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.581225 8859 raft_consensus.cc:374] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:04.581264 8859 raft_consensus.cc:729] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9423d1ffdd4ae89ba67d35bc188b2e, State: Initialized, Role: FOLLOWER
I20260926 09:25:04.581372 8859 consensus_queue.cc:260] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.581418 8861 ts_tablet_manager.cc:1428] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 09:25:04.581465 8842 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
I20260926 09:25:04.581634 8859 ts_tablet_manager.cc:1428] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:25:04.581677 8708 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
W20260926 09:25:04.623076 8709 tablet.cc:2369] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:25:04.677259 8866 raft_consensus.cc:482] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:04.677446 8866 raft_consensus.cc:504] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.677856 8866 leader_election.cc:288] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
I20260926 09:25:04.681945 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.678167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:43369 (local address 127.24.235.66:36835)
0926 09:25:04.678250 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.678252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.678285 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:25:04.678318 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.678320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.678333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.678400 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:04.678565 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.679361 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.680957 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.681556 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.681638 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.681646 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.681830 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.681869 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 09:25:04.681945 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.678170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:40661 (local address 127.24.235.65:44609)
0926 09:25:04.678247 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.678250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.678319 (+ 69us) server_negotiation.cc:407] Connection header received
0926 09:25:04.678393 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.678394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.678406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.678458 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:04.678626 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.679368 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.680957 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.681560 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.681637 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.681646 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.681830 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.681869 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:25:04.681991 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:36835 (local address 127.24.235.67:43369)
0926 09:25:04.678156 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.678170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.678239 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.678426 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.678427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.678429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:04.678506 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.678508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.679463 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.679463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.680852 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.680856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.681650 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.681651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.681747 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.681751 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.681814 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.681877 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":126}
I20260926 09:25:04.682241 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.24.235.67:40661)
0926 09:25:04.678261 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.678271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:04.678331 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:04.678482 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:04.678483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:04.678485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:04.678547 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.678549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.679462 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.679463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.680857 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:04.680859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.681686 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.681687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:04.681748 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.681751 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:25:04.681814 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:25:04.681876 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":111,"thread_start_us":116,"threads_started":1}
I20260926 09:25:04.682343 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf2b30b9be46f79eb2b3db2bdb15bd" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" is_pre_election: true
I20260926 09:25:04.682529 8662 raft_consensus.cc:2454] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 0.
I20260926 09:25:04.682682 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf2b30b9be46f79eb2b3db2bdb15bd" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" is_pre_election: true
I20260926 09:25:04.682781 8730 leader_election.cc:302] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e; no voters:
I20260926 09:25:04.682863 8528 raft_consensus.cc:2454] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 0.
I20260926 09:25:04.682915 8866 raft_consensus.cc:2781] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:25:04.682986 8866 raft_consensus.cc:482] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:25:04.683023 8866 raft_consensus.cc:3037] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:04.683827 8866 raft_consensus.cc:504] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.683991 8866 leader_election.cc:288] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 election: Requested vote from peers d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
I20260926 09:25:04.684152 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf2b30b9be46f79eb2b3db2bdb15bd" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
I20260926 09:25:04.684161 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dbf2b30b9be46f79eb2b3db2bdb15bd" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
I20260926 09:25:04.684242 8662 raft_consensus.cc:3037] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:04.684242 8528 raft_consensus.cc:3037] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:04.685297 8528 raft_consensus.cc:2454] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 1.
I20260926 09:25:04.685297 8662 raft_consensus.cc:2454] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 1.
I20260926 09:25:04.685485 8730 leader_election.cc:302] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e; no voters:
I20260926 09:25:04.685600 8866 raft_consensus.cc:2781] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:25:04.685782 8866 raft_consensus.cc:686] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 LEADER]: Becoming Leader. State: Replica: 0a409cd42ddc4ae8a7e9f68ad55b78db, State: Running, Role: LEADER
I20260926 09:25:04.685883 8866 consensus_queue.cc:237] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:04.686724 8381 catalog_manager.cc:5495] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67). New cstate: current_term: 1 leader_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } health_report { overall_health: UNKNOWN } } }
W20260926 09:25:04.701717 8575 tablet.cc:2369] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:25:04.718247 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.715660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.24.235.67:34459)
0926 09:25:04.715733 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:04.715735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:04.715748 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:25:04.715786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:04.715787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:04.715804 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:04.715899 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:25:04.716040 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.716513 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.717439 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:04.717636 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:04.717704 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:04.717891 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:25:04.717992 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:25:04.718094 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:04.718111 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 09:25:04.718132 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:25:04.720350 8662 raft_consensus.cc:1261] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Refusing update from remote peer 0a409cd42ddc4ae8a7e9f68ad55b78db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:25:04.720350 8528 raft_consensus.cc:1261] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Refusing update from remote peer 0a409cd42ddc4ae8a7e9f68ad55b78db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:25:04.720755 8866 consensus_queue.cc:1035] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:25:04.720924 8869 consensus_queue.cc:1035] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:25:04.724090 8874 mvcc.cc:205] Tried to move back new op lower bound from 7333538630531620864 to 7333538630394064896. Current Snapshot: MvccSnapshot[applied={T|T < 7333538630531620864}]
I20260926 09:25:04.724661 8873 mvcc.cc:205] Tried to move back new op lower bound from 7333538630531620864 to 7333538630394064896. Current Snapshot: MvccSnapshot[applied={T|T < 7333538630531620864 or (T in {7333538630531620864})}]
I20260926 09:25:04.725553 8381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:25:04.725666 8381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:25:04.726534 8381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Sending DeleteTablet for 3 replicas of tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd
I20260926 09:25:04.726903 8776 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:04 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:60390
I20260926 09:25:04.726918 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:04 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:45288
I20260926 09:25:04.727090 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8445
I20260926 09:25:04.727190 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:04 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:48118
I20260926 09:25:04.729148 8883 tablet_replica.cc:330] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: stopping tablet replica
I20260926 09:25:04.729280 8883 raft_consensus.cc:2229] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:25:04.729421 8883 raft_consensus.cc:2258] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:25:04.729903 8883 ts_tablet_manager.cc:1905] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:25:04.732093 8883 ts_tablet_manager.cc:1918] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:25:04.732214 8883 log.cc:1196] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/9dbf2b30b9be46f79eb2b3db2bdb15bd
I20260926 09:25:04.732534 8883 ts_tablet_manager.cc:1939] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting consensus metadata
I20260926 09:25:04.733177 8882 tablet_replica.cc:330] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: stopping tablet replica
I20260926 09:25:04.733256 8367 catalog_manager.cc:4841] TS cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (table test-table [id=7791410430d0430cb9767d4fc6071023]) successfully deleted
I20260926 09:25:04.733312 8882 raft_consensus.cc:2229] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:25:04.733422 8882 raft_consensus.cc:2258] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:25:04.733990 8882 ts_tablet_manager.cc:1905] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 09:25:04.734458 8369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:44609 (error 108)
I20260926 09:25:04.734733 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8579
I20260926 09:25:04.735878 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:04.735726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.0.0.1:45294)
0926 09:25:04.735781 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:04.735839 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 09:25:04.736019 8369 catalog_manager.cc:4584] TS d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44609: connect: Connection refused (error 111)
I20260926 09:25:04.737712 8882 ts_tablet_manager.cc:1918] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:25:04.737797 8882 log.cc:1196] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/9dbf2b30b9be46f79eb2b3db2bdb15bd
I20260926 09:25:04.738116 8882 ts_tablet_manager.cc:1939] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting consensus metadata
I20260926 09:25:04.738605 8368 catalog_manager.cc:4841] TS 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (table test-table [id=7791410430d0430cb9767d4fc6071023]) successfully deleted
I20260926 09:25:04.740245 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8713
I20260926 09:25:04.746429 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8350
I20260926 09:25:24.752749 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--webserver_interface=127.24.235.126
--webserver_port=42923
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:24.837271 8885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:24.837503 8885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:24.837532 8885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:24.838786 8885 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:25:24.838833 8885 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:25:24.838847 8885 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:25:24.838860 8885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:25:24.838872 8885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:24.838886 8885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:24.838898 8885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:25:24.838910 8885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:25:24.840435 8885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:24.840669 8885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:24.840858 8885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:24.851513 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:24.851511 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:24.851555 8885 server_base.cc:1034] running on GCE node
W20260926 09:25:24.851516 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:24.852082 8885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:24.852487 8885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:24.853703 8885 hybrid_clock.cc:648] HybridClock initialized: now 1790414724853683 us; error 47 us; skew 500 ppm
Sep 26 09:25:24 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414724, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:24.856346 8885 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:25:24.856827 8885 webserver.cc:460] Webserver started at http://127.24.235.126:42923/ using document root <none> and password file <none>
I20260926 09:25:24.857141 8885 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:24.857204 8885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:24.858727 8885 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:24.859444 8901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:24.859692 8885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:24.859797 8885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "7cb09fd09a0d4ed88caa0d798b7525f1"
format_stamp: "Formatted at 2026-09-26 09:25:03 on dist-test-slave-7zlw"
server_key: "a78577027cce75f492aba3734a606384"
server_key_iv: "3b05d95026c45a6a7b4a1c23605158e0"
server_key_version: "encryptionkey@0"
I20260926 09:25:24.860221 8885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:24.872025 8885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:24.874645 8885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:24.874773 8885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:24.879529 8885 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41015
I20260926 09:25:24.879556 8953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41015 every 8 connection(s)
I20260926 09:25:24.879978 8885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:25:24.881983 8954 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:25:24.882683 8954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap starting.
I20260926 09:25:24.889094 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8885
I20260926 09:25:24.889467 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:44609
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=40463
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:25:24.892395 8954 log.cc:824] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:24.894554 8954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:25:24.894897 8954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap complete.
I20260926 09:25:24.897374 8954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:24.897604 8954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Initialized, Role: FOLLOWER
I20260926 09:25:24.897774 8954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:24.897841 8954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:25:24.897907 8954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:25:24.897953 8954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 09:25:24.898994 8954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:24.899083 8954 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb09fd09a0d4ed88caa0d798b7525f1; no voters:
I20260926 09:25:24.899333 8954 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 09:25:24.899363 8958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 09:25:24.899484 8958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 LEADER]: Becoming Leader. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Running, Role: LEADER
I20260926 09:25:24.899622 8954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:25:24.899638 8958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:24.900014 8959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:24.900032 8960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb09fd09a0d4ed88caa0d798b7525f1. Latest consensus state: current_term: 2 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:24.900113 8959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:24.900139 8960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:24.900365 8964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:25:24.901324 8964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:25:24.901476 8964 catalog_manager.cc:1259] Loaded cluster ID: dd5c411e29ea4087905f8ee9e6ba8a9c
I20260926 09:25:24.901719 8964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:25:24.902878 8964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:25:24.903059 8964 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Loaded TSK: 0
I20260926 09:25:24.914342 8964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Generated new TSK 1
I20260926 09:25:24.914451 8964 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:25:24.974090 8956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:24.974341 8956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:24.974381 8956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:24.975526 8956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:24.975593 8956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:24.975637 8956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:25:24.977151 8956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:44609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=40463
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:24.977411 8956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:24.977643 8956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:24.984172 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:24.984179 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:24.984215 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:24.984615 8956 server_base.cc:1034] running on GCE node
I20260926 09:25:24.984874 8956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:24.985222 8956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:24.986436 8956 hybrid_clock.cc:648] HybridClock initialized: now 1790414724986424 us; error 41 us; skew 500 ppm
Sep 26 09:25:24 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414724, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:24.989774 8956 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:25:24.990294 8956 webserver.cc:460] Webserver started at http://127.24.235.65:40463/ using document root <none> and password file <none>
I20260926 09:25:24.990525 8956 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:24.990612 8956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:24.992075 8956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:24.992889 8991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:24.993171 8956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:24.993284 8956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "f0f0b4775ea412d769943caf5d3e3f7b"
server_key_iv: "c2ecda30c216e1085965a2a222f00f3e"
server_key_version: "encryptionkey@0"
I20260926 09:25:24.993665 8956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:25.009723 8956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:25.012347 8956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:25.012523 8956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:25.012848 8956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:25.013865 8998 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:25:25.016027 8956 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:25:25.016101 8956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260926 09:25:25.016156 8956 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:25:25.016827 8956 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:25:25.016886 8956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:25:25.016978 8998 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap starting.
I20260926 09:25:25.024324 8956 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44609
I20260926 09:25:25.024693 8956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:25:25.025432 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 8956
I20260926 09:25:25.025763 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:36835
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=46801
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:25:25.027428 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44609 every 8 connection(s)
I20260926 09:25:25.029881 8998 log.cc:824] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Log is configured to *not* fsync() on all Append() calls
Sep 26 09:25:25 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414724, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:25.036746 8998 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:25:25.037456 8998 tablet_bootstrap.cc:492] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap complete.
I20260926 09:25:25.038812 8998 ts_tablet_manager.cc:1397] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.010s
I20260926 09:25:25.040416 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.025826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:49241 (local address 127.24.235.126:41015)
0926 09:25:25.026246 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.026257 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.027309 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 09:25:25.027453 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.027460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.027572 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.027722 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:25.028608 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.030127 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.031831 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.032527 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.032626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.035955 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:25.035960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:25.035961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:25.035974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:25.038368 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.038906 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.038910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.038910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.039027 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.039331 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.039332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.039333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.039607 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:25.039661 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.040046 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.040201 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":71,"threads_started":1}
I20260926 09:25:25.040133 8998 raft_consensus.cc:348] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:25.040465 8998 raft_consensus.cc:729] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c110a2424844b7a50e3a9e6531b9cd, State: Initialized, Role: FOLLOWER
I20260926 09:25:25.040637 8998 consensus_queue.cc:260] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } }
I20260926 09:25:25.040892 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.025423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:49241)
0926 09:25:25.026346 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.026393 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.027138 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.027747 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.027752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.028157 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:25.028462 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.028469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.030287 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.030289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.031718 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.031722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.032781 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.032782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.032938 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.033726 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:25.033745 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:25.035790 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:25.038467 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.038473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.038476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.038802 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.039117 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.039118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.039118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.039255 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.039716 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:25.039722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:25.039867 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.040525 (+ 658us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.040696 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":100,"threads_started":1}
I20260926 09:25:25.040999 8998 ts_tablet_manager.cc:1428] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260926 09:25:25.041731 9106 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:25.041826 9106 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:25.041996 9106 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:25.042793 8918 ts_manager.cc:194] Registered new tserver with Master: d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609)
I20260926 09:25:25.043994 8918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:49241
I20260926 09:25:25.044956 9106 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
I20260926 09:25:25.053282 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.044803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.24.235.65:44609)
0926 09:25:25.048977 (+ 4174us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.048981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.048992 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:25:25.049066 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.049068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.049083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.049172 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:25.049429 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.050509 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.052110 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.052783 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.052860 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.052914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.053128 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.053203 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4137,"thread_start_us":93,"threads_started":1}
I20260926 09:25:25.053305 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.044430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.0.0.1:58940)
0926 09:25:25.044987 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.045135 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.045205 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.049217 (+ 4012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.049222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.049226 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:25.049323 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.049326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.050616 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.050617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.051948 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.051951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.052874 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.052875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.052967 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.053004 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.053153 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.053215 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":125,"threads_started":1}
I20260926 09:25:25.053926 9029 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:04 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:58940
I20260926 09:25:25.054571 9115 tablet_replica.cc:330] stopping tablet replica
I20260926 09:25:25.054688 9115 raft_consensus.cc:2229] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:25:25.054749 9115 raft_consensus.cc:2258] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:25:25.055054 9115 ts_tablet_manager.cc:1905] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:25:25.057597 9115 ts_tablet_manager.cc:1918] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:25:25.057686 9115 log.cc:1196] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal/wals/9dbf2b30b9be46f79eb2b3db2bdb15bd
I20260926 09:25:25.057910 9115 ts_tablet_manager.cc:1939] T 9dbf2b30b9be46f79eb2b3db2bdb15bd P d6c110a2424844b7a50e3a9e6531b9cd: Deleting consensus metadata
I20260926 09:25:25.058472 8905 catalog_manager.cc:4841] TS d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609): tablet 9dbf2b30b9be46f79eb2b3db2bdb15bd (table test-table [id=7791410430d0430cb9767d4fc6071023]) successfully deleted
W20260926 09:25:25.117623 9110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:25.117830 9110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:25.117853 9110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:25.118935 9110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:25.118968 9110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:25.118992 9110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:25:25.120364 9110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:36835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=46801
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:25.120589 9110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:25.120774 9110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:25.125906 9121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:25.125945 9123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:25.125945 9120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:25.126221 9110 server_base.cc:1034] running on GCE node
I20260926 09:25:25.126427 9110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:25.126617 9110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:25.127759 9110 hybrid_clock.cc:648] HybridClock initialized: now 1790414725127748 us; error 28 us; skew 500 ppm
Sep 26 09:25:25 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414725, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:25.130275 9110 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:25:25.130718 9110 webserver.cc:460] Webserver started at http://127.24.235.66:46801/ using document root <none> and password file <none>
I20260926 09:25:25.130911 9110 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:25.130960 9110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:25.132232 9110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:25.132817 9130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.132987 9110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:25.133116 9110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "365d262b78c01f4139340761fd1da777"
server_key_iv: "f5e9babee6d4b01bd8066d94969c938a"
server_key_version: "encryptionkey@0"
I20260926 09:25:25.133481 9110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:25.147884 9110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:25.150267 9110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:25.150435 9110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:25.150671 9110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:25.150995 9110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:25.151049 9110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.151090 9110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:25.151118 9110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.157577 9110 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:36835
I20260926 09:25:25.157635 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:36835 every 8 connection(s)
I20260926 09:25:25.158027 9110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:25:25.162765 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9110
I20260926 09:25:25.163131 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:34459
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=38797
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 09:25:25 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414725, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:25.168576 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.158878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:49769 (local address 127.24.235.126:41015)
0926 09:25:25.158966 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.158968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.159210 (+ 242us) server_negotiation.cc:407] Connection header received
0926 09:25:25.159832 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.159833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.159844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.159896 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:25.160920 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.161497 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.162464 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.162656 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.162742 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.165302 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:25.165304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:25.165304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:25.165313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:25.167065 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.167478 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.167480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.167480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.167544 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.167803 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.167804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.167804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.167967 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:25.168046 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.168335 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.168438 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:25:25.168587 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.158825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:49769)
0926 09:25:25.159039 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.159089 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.159720 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.159978 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.159982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.160622 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:25.160824 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.160829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.161576 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.161577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.162391 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.162393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.162753 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.162754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.162952 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.163483 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:25.163491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:25.165205 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:25.167156 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.167163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.167165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.167416 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.167625 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.167626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.167627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.167731 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.168039 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:25.168044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:25.168180 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.168283 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.168412 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260926 09:25:25.169183 9244 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:25.169301 9244 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:25.169521 9244 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:25.169982 8918 ts_manager.cc:194] Registered new tserver with Master: cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
I20260926 09:25:25.170380 8918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:49769
W20260926 09:25:25.249337 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:25.249539 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:25.249563 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:25.250833 9247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:25.250878 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:25.250905 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:25:25.252395 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:34459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=38797
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:25.252709 9247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:25.252960 9247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:25.258358 9252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:25.258371 9253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:25.258571 9247 server_base.cc:1034] running on GCE node
W20260926 09:25:25.258411 9255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:25.258929 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:25.259209 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:25.260414 9247 hybrid_clock.cc:648] HybridClock initialized: now 1790414725260408 us; error 44 us; skew 500 ppm
Sep 26 09:25:25 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414725, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:25.263417 9247 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:25:25.263963 9247 webserver.cc:460] Webserver started at http://127.24.235.67:38797/ using document root <none> and password file <none>
I20260926 09:25:25.264226 9247 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:25.264353 9247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:25.265970 9247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:25.266639 9262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.266920 9247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:25.267012 9247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "e5b064c4efadbfdd7574f516402224ec"
server_key_iv: "e61b0361dfab786c81d057d29273bf74"
server_key_version: "encryptionkey@0"
I20260926 09:25:25.267350 9247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:25.277348 9247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:25.279537 9247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:25.279651 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:25.279824 9247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:25.280079 9247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:25.280109 9247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.280131 9247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:25.280143 9247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:25.285933 9247 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:34459
I20260926 09:25:25.286015 9375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:34459 every 8 connection(s)
I20260926 09:25:25.286365 9247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:25:25.289461 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9247
Sep 26 09:25:25 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414725, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:25:25.298012 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.290029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.24.235.126:41015)
0926 09:25:25.290187 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.290189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.290279 (+ 90us) server_negotiation.cc:407] Connection header received
0926 09:25:25.290308 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.290309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.290320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.290395 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:25.291088 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.291733 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.292685 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.293010 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.293130 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.294397 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:25.294398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:25.294398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:25.294405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:25.296589 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.297065 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.297066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.297066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.297134 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.297406 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.297407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.297407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.297538 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:25.297724 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.297866 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.297938 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260926 09:25:25.298326 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.287152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:46291 (local address 127.24.235.126:41015)
0926 09:25:25.287222 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.287224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.287455 (+ 231us) server_negotiation.cc:407] Connection header received
0926 09:25:25.288008 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.288010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.288022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.288068 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:25.289045 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.289906 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.291146 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.291348 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.291401 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.294314 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:25.294315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:25.294316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:25.294323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:25.296488 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.297176 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.297177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.297177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.297233 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.297542 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.297543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.297543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.297668 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:25.297734 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.298141 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.298235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:25:25.298830 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.287154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:46291)
0926 09:25:25.287336 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.287372 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.287890 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.288143 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.288152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.288729 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:25.288917 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.288922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.290041 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.290043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.291057 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.291061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.291432 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.291433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.291576 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.292214 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:25.292222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:25.294199 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:25.296592 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.296599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.296602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.297091 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.297316 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:25.297317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:25.297317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:25.297463 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:25.297761 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:25.297769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:25.297934 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.298460 (+ 526us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.298663 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260926 09:25:25.299355 9376 heartbeater.cc:347] Connected to a master server at 127.24.235.126:41015
I20260926 09:25:25.299456 9376 heartbeater.cc:464] Registering TS with master...
I20260926 09:25:25.299698 9376 heartbeater.cc:511] Master 127.24.235.126:41015 requested a full tablet report, sending...
I20260926 09:25:25.300146 8918 ts_manager.cc:194] Registered new tserver with Master: 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459)
I20260926 09:25:25.300549 8918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:46291
I20260926 09:25:25.309968 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:25:25.312806 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.310381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.24.235.126:41015)
0926 09:25:25.310440 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.310441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.310589 (+ 148us) server_negotiation.cc:407] Connection header received
0926 09:25:25.310691 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.310692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.310728 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.310797 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:25:25.310970 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.311438 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.312163 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.312347 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.312401 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.312578 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:25:25.312621 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 09:25:25.312750 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33828: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 09:25:25.312886 9109 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33828: token expired
W20260926 09:25:25.313031 8846 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.235.126:41015)
I20260926 09:25:25.319646 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.313407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.24.235.126:41015)
0926 09:25:25.313475 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.313477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.313485 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:25:25.313504 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.313505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.313513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.313588 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:25.314120 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.314624 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.315391 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.315616 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.315698 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.316940 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:25.316942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:25.316943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:25.316951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:25.318367 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.318796 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.318797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.318797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.318856 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:25.319070 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:25.319071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:25.319072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:25.319230 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:25.319337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.319462 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.319542 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:25:25.320817 8918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:25:25.322700 8918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:25:25.324928 9029 tablet_service.cc:1467] Processing CreateTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 (DEFAULT_TABLE table=test-table [id=4dd801bc46024901837f9c409167cd9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:25.325194 9029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79037f1e7d9741ab9c20cf5bf3b13321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:25.326824 8998 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap starting.
I20260926 09:25:25.327199 8998 tablet_bootstrap.cc:654] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:25.327909 8998 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: No bootstrap required, opened a new log
I20260926 09:25:25.327966 8998 ts_tablet_manager.cc:1397] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:25.328094 8998 raft_consensus.cc:348] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.328181 8998 raft_consensus.cc:374] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:25.328238 8998 raft_consensus.cc:729] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c110a2424844b7a50e3a9e6531b9cd, State: Initialized, Role: FOLLOWER
I20260926 09:25:25.328305 8998 consensus_queue.cc:260] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.328408 8998 ts_tablet_manager.cc:1428] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 09:25:25.329077 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.324616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:36835 (local address 127.0.0.1:52068)
0926 09:25:25.324664 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.324674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.324711 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.325366 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.325367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.325369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:25.325415 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.325417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.326890 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.326891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.327857 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.327860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.328777 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.328778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.328846 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.328851 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.328916 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.328967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 09:25:25.329264 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.324842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.24.235.66:36835)
0926 09:25:25.325165 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.325169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.325178 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:25:25.325220 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.325222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.325238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.325362 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:25.325490 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.326706 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.327942 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.328687 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.328765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.328829 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.329080 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.329157 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":100,"threads_started":1}
I20260926 09:25:25.329308 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.324626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34459 (local address 127.0.0.1:49404)
0926 09:25:25.325006 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.325039 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.325095 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.325417 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.325417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.325420 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:25.325457 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.325459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.326522 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.326523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.327931 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.327934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.329054 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.329055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.329120 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.329124 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.329197 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.329237 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":125,"threads_started":1}
I20260926 09:25:25.329452 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.324855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.24.235.67:34459)
0926 09:25:25.325063 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.325068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.325084 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:25:25.325200 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.325203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.325220 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.325323 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:25.325560 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.326427 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.328039 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.328882 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.328946 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.329000 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.329293 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.329341 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260926 09:25:25.329957 9178 tablet_service.cc:1467] Processing CreateTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 (DEFAULT_TABLE table=test-table [id=4dd801bc46024901837f9c409167cd9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:25.330086 9310 tablet_service.cc:1467] Processing CreateTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 (DEFAULT_TABLE table=test-table [id=4dd801bc46024901837f9c409167cd9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:25:25.330315 9178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79037f1e7d9741ab9c20cf5bf3b13321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:25.330371 9310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79037f1e7d9741ab9c20cf5bf3b13321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:25:25.333853 9387 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Bootstrap starting.
I20260926 09:25:25.333853 9386 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Bootstrap starting.
I20260926 09:25:25.334622 9386 tablet_bootstrap.cc:654] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:25.334615 9387 tablet_bootstrap.cc:654] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Neither blocks nor log segments found. Creating new log.
I20260926 09:25:25.335328 9387 log.cc:824] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:25.335328 9386 log.cc:824] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:25.336166 9386 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: No bootstrap required, opened a new log
I20260926 09:25:25.336253 9386 ts_tablet_manager.cc:1397] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:25:25.336555 9387 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: No bootstrap required, opened a new log
I20260926 09:25:25.336632 9387 ts_tablet_manager.cc:1397] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:25:25.338519 9386 raft_consensus.cc:348] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.338725 9386 raft_consensus.cc:374] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:25.338768 9386 raft_consensus.cc:729] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9423d1ffdd4ae89ba67d35bc188b2e, State: Initialized, Role: FOLLOWER
I20260926 09:25:25.338819 9387 raft_consensus.cc:348] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.338909 9386 consensus_queue.cc:260] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.338989 9387 raft_consensus.cc:374] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:25:25.339039 9387 raft_consensus.cc:729] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a409cd42ddc4ae8a7e9f68ad55b78db, State: Initialized, Role: FOLLOWER
I20260926 09:25:25.339174 9387 consensus_queue.cc:260] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.339246 9244 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
I20260926 09:25:25.339461 9387 ts_tablet_manager.cc:1428] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:25:25.339550 9376 heartbeater.cc:503] Master 127.24.235.126:41015 was elected leader, sending a full tablet report...
I20260926 09:25:25.339239 9386 ts_tablet_manager.cc:1428] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 09:25:25.408720 9245 tablet.cc:2369] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:25:25.433856 9391 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:25.433997 9391 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.434485 9391 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835), d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609)
I20260926 09:25:25.437736 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.434819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:55821 (local address 127.24.235.65:44609)
0926 09:25:25.434880 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.434882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.434895 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:25:25.434929 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.434930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.434941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.435002 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:25.435214 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.435784 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.436862 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.437361 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.437419 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.437426 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.437621 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.437664 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 09:25:25.437783 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.434840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:60105 (local address 127.24.235.66:36835)
0926 09:25:25.434897 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.434900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.434964 (+ 64us) server_negotiation.cc:407] Connection header received
0926 09:25:25.435022 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.435023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.435037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.435105 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:25:25.435307 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.435859 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.436969 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.437435 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.437506 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.437516 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.437681 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.437720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:25:25.437783 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.434687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.24.235.67:55821)
0926 09:25:25.434757 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.434772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.434842 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.435019 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.435039 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.435041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:25.435122 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.435125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.435861 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.435862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.436777 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.436779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.437445 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.437446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.437533 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.437537 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.437590 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.437653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":5}
I20260926 09:25:25.437785 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.434709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:36835 (local address 127.24.235.67:60105)
0926 09:25:25.434882 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.434892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:25.434958 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:25.435113 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:25.435114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:25.435115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:25:25.435203 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.435205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.435965 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.435966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.436884 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:25.436886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.437512 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.437513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:25.437566 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.437569 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:25:25.437654 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:25:25.437688 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260926 09:25:25.438180 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79037f1e7d9741ab9c20cf5bf3b13321" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" is_pre_election: true
I20260926 09:25:25.438216 9198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79037f1e7d9741ab9c20cf5bf3b13321" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" is_pre_election: true
I20260926 09:25:25.438355 9055 raft_consensus.cc:2454] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 0.
I20260926 09:25:25.438369 9198 raft_consensus.cc:2454] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 0.
I20260926 09:25:25.438585 9266 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, d6c110a2424844b7a50e3a9e6531b9cd; no voters:
I20260926 09:25:25.438735 9391 raft_consensus.cc:2781] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:25:25.438817 9391 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:25:25.438853 9391 raft_consensus.cc:3037] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:25.439568 9391 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.439752 9391 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 election: Requested vote from peers cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835), d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609)
I20260926 09:25:25.439898 9198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79037f1e7d9741ab9c20cf5bf3b13321" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
I20260926 09:25:25.439913 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79037f1e7d9741ab9c20cf5bf3b13321" candidate_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
I20260926 09:25:25.439980 9055 raft_consensus.cc:3037] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:25.439980 9198 raft_consensus.cc:3037] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:25:25.440936 9198 raft_consensus.cc:2454] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 1.
I20260926 09:25:25.441129 9264 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e; no voters:
I20260926 09:25:25.441242 9391 raft_consensus.cc:2781] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:25:25.441407 9391 raft_consensus.cc:686] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 LEADER]: Becoming Leader. State: Replica: 0a409cd42ddc4ae8a7e9f68ad55b78db, State: Running, Role: LEADER
I20260926 09:25:25.441392 9055 raft_consensus.cc:2454] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a409cd42ddc4ae8a7e9f68ad55b78db in term 1.
I20260926 09:25:25.441522 9391 consensus_queue.cc:237] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:25.442385 8918 catalog_manager.cc:5495] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67). New cstate: current_term: 1 leader_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } health_report { overall_health: UNKNOWN } } }
I20260926 09:25:25.448261 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.446076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.24.235.67:34459)
0926 09:25:25.446131 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:25.446133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:25.446140 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:25:25.446195 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:25.446196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:25.446211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:25.446268 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:25:25.446380 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.446820 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.447575 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:25.447774 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:25.447826 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:25.447948 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:25:25.448039 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:25:25.448098 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:25.448130 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 09:25:25.448145 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:25:25.450131 9055 raft_consensus.cc:1261] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Refusing update from remote peer 0a409cd42ddc4ae8a7e9f68ad55b78db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:25:25.450169 9198 raft_consensus.cc:1261] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Refusing update from remote peer 0a409cd42ddc4ae8a7e9f68ad55b78db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:25:25.450505 9393 consensus_queue.cc:1035] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:25:25.450629 9391 consensus_queue.cc:1035] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:25:25.452998 9395 mvcc.cc:205] Tried to move back new op lower bound from 7333538715441004544 to 7333538715409326080. Current Snapshot: MvccSnapshot[applied={T|T < 7333538715441004544}]
I20260926 09:25:25.453317 9396 mvcc.cc:205] Tried to move back new op lower bound from 7333538715441004544 to 7333538715409326080. Current Snapshot: MvccSnapshot[applied={T|T < 7333538715441004544}]
I20260926 09:25:25.455190 8917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:25:25.455292 8917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:25:25.455454 9398 mvcc.cc:205] Tried to move back new op lower bound from 7333538715441004544 to 7333538715409326080. Current Snapshot: MvccSnapshot[applied={T|T < 7333538715441004544}]
I20260926 09:25:25.456226 8917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Sending DeleteTablet for 3 replicas of tablet 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:25.456409 9310 tablet_service.cc:1514] Processing DeleteTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:25 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:49404
I20260926 09:25:25.456550 9029 tablet_service.cc:1514] Processing DeleteTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:25 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:58940
I20260926 09:25:25.456549 9178 tablet_service.cc:1514] Processing DeleteTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:25 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:52068
I20260926 09:25:25.456641 9115 tablet_replica.cc:330] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: stopping tablet replica
I20260926 09:25:25.456682 9115 raft_consensus.cc:2229] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:25:25.456724 9040 tablet_service.cc:1514] Processing DeleteTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:25 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:58940
I20260926 09:25:25.456831 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8956
I20260926 09:25:25.456871 9405 tablet_replica.cc:330] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: stopping tablet replica
I20260926 09:25:25.457815 9406 tablet_replica.cc:330] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: stopping tablet replica
I20260926 09:25:25.457928 9406 raft_consensus.cc:2229] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:25:25.458043 9406 raft_consensus.cc:2258] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:25:25.458542 9406 ts_tablet_manager.cc:1905] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:25:25.457617 9310 tablet_service.cc:1514] Processing DeleteTablet for tablet 79037f1e7d9741ab9c20cf5bf3b13321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:25:25 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:49404
I20260926 09:25:25.460223 9405 raft_consensus.cc:2229] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:25:25.460354 9405 raft_consensus.cc:2258] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:25:25.460749 9405 ts_tablet_manager.cc:1905] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:25:25.460873 9406 ts_tablet_manager.cc:1918] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:25:25.460949 9406 log.cc:1196] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:25.461293 9406 ts_tablet_manager.cc:1939] T 79037f1e7d9741ab9c20cf5bf3b13321 P cf9423d1ffdd4ae89ba67d35bc188b2e: Deleting consensus metadata
I20260926 09:25:25.461779 8903 catalog_manager.cc:4841] TS cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): tablet 79037f1e7d9741ab9c20cf5bf3b13321 (table test-table [id=4dd801bc46024901837f9c409167cd9b]) successfully deleted
I20260926 09:25:25.462831 9405 ts_tablet_manager.cc:1918] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:25:25.462890 9405 log.cc:1196] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:25.463176 9405 ts_tablet_manager.cc:1939] T 79037f1e7d9741ab9c20cf5bf3b13321 P 0a409cd42ddc4ae8a7e9f68ad55b78db: Deleting consensus metadata
I20260926 09:25:25.463662 8904 catalog_manager.cc:4841] TS 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): tablet 79037f1e7d9741ab9c20cf5bf3b13321 (table test-table [id=4dd801bc46024901837f9c409167cd9b]) successfully deleted
W20260926 09:25:25.463805 8904 catalog_manager.cc:4808] TS 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): delete failed for tablet 79037f1e7d9741ab9c20cf5bf3b13321 because the tablet was not found. No further retry: Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:25:25.468675 8905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:44609 (error 108)
I20260926 09:25:25.468989 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9110
I20260926 09:25:25.469264 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:25.469130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44609 (local address 127.0.0.1:58946)
0926 09:25:25.469187 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:25.469224 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:25:25.469368 8905 catalog_manager.cc:4584] TS d6c110a2424844b7a50e3a9e6531b9cd (127.24.235.65:44609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79037f1e7d9741ab9c20cf5bf3b13321: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44609: connect: Connection refused (error 111)
I20260926 09:25:25.474814 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9247
I20260926 09:25:25.480118 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 8885
I20260926 09:25:45.486414 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--webserver_interface=127.24.235.126
--webserver_port=42923
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:45.569253 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:45.569509 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:45.569535 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:45.570650 9408 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 09:25:45.570683 9408 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 09:25:45.570698 9408 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 09:25:45.570709 9408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:25:45.570722 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:45.570736 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:45.570748 9408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:25:45.570760 9408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:25:45.572278 9408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=42923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:45.572477 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:45.572660 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:45.578150 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:45.578253 9408 server_base.cc:1034] running on GCE node
W20260926 09:25:45.578334 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:45.578346 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:45.578785 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:45.579155 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:45.580363 9408 hybrid_clock.cc:648] HybridClock initialized: now 1790414745580341 us; error 50 us; skew 500 ppm
Sep 26 09:25:45 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414745, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:45.583381 9408 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:25:45.583858 9408 webserver.cc:460] Webserver started at http://127.24.235.126:42923/ using document root <none> and password file <none>
I20260926 09:25:45.584091 9408 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:45.584144 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:45.585758 9408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 09:25:45.586575 9423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:45.586881 9408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:25:45.586987 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "7cb09fd09a0d4ed88caa0d798b7525f1"
format_stamp: "Formatted at 2026-09-26 09:25:03 on dist-test-slave-7zlw"
server_key: "a78577027cce75f492aba3734a606384"
server_key_iv: "3b05d95026c45a6a7b4a1c23605158e0"
server_key_version: "encryptionkey@0"
I20260926 09:25:45.587405 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:45.595676 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:45.598155 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:45.598315 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:45.602779 9408 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:41015
I20260926 09:25:45.602862 9476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:41015 every 8 connection(s)
I20260926 09:25:45.603240 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:25:45.604848 9477 sys_catalog.cc:263] Verifying existing consensus state
I20260926 09:25:45.605626 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap starting.
I20260926 09:25:45.612526 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9408
I20260926 09:25:45.612917 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:44609
--local_ip_for_outbound_sockets=127.24.235.65
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=40463
--webserver_interface=127.24.235.65
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:25:45.616065 9477 log.cc:824] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:45.618816 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 09:25:45.619164 9477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Bootstrap complete.
I20260926 09:25:45.621987 9477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:45.622256 9477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Initialized, Role: FOLLOWER
I20260926 09:25:45.622480 9477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:45.622588 9477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:25:45.622648 9477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:25:45.622711 9477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 2 FOLLOWER]: Advancing to term 3
I20260926 09:25:45.623795 9477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:45.623927 9477 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb09fd09a0d4ed88caa0d798b7525f1; no voters:
I20260926 09:25:45.624162 9477 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 09:25:45.624217 9481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 09:25:45.624361 9481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [term 3 LEADER]: Becoming Leader. State: Replica: 7cb09fd09a0d4ed88caa0d798b7525f1, State: Running, Role: LEADER
I20260926 09:25:45.624464 9477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:25:45.624457 9481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } }
I20260926 09:25:45.625419 9483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb09fd09a0d4ed88caa0d798b7525f1. Latest consensus state: current_term: 3 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:45.625540 9483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:45.625756 9491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:25:45.625926 9482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb09fd09a0d4ed88caa0d798b7525f1" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 41015 } } }
I20260926 09:25:45.625998 9482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1 [sys.catalog]: This master's current role is: LEADER
I20260926 09:25:45.626749 9491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:25:45.627007 9491 catalog_manager.cc:1259] Loaded cluster ID: dd5c411e29ea4087905f8ee9e6ba8a9c
I20260926 09:25:45.627106 9491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:25:45.627971 9491 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:25:45.628088 9491 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Loaded TSK: 0
I20260926 09:25:45.628113 9491 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cb09fd09a0d4ed88caa0d798b7525f1: Loaded TSK: 1
F20260926 09:25:45.628304 9491 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f932dd0c9ad google::LogMessage::Fail() at ??:0
@ 0x7f932dd0f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f932dd0c4de google::LogMessage::Flush() at ??:0
@ 0x7f932dd0da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9330479104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9330476d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9335c661cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9335c6648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9335c66594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9335c2cc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9335c36ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f932fbda0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f932fbcf954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9330a336db start_thread at ??:0
@ 0x7f932c80a61f clone at ??:0
W20260926 09:25:45.719408 9479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:45.719625 9479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:45.719652 9479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:45.720837 9479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:45.720885 9479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:45.720914 9479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:25:45.722451 9479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:44609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=40463
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:45.722704 9479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:45.722973 9479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:45.729898 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:45.729902 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:45.730065 9479 server_base.cc:1034] running on GCE node
W20260926 09:25:45.729902 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:45.730571 9479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:45.730882 9479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:45.732079 9479 hybrid_clock.cc:648] HybridClock initialized: now 1790414745732050 us; error 43 us; skew 500 ppm
Sep 26 09:25:45 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414745, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:45.735632 9479 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:25:45.736310 9479 webserver.cc:460] Webserver started at http://127.24.235.65:40463/ using document root <none> and password file <none>
I20260926 09:25:45.736640 9479 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:45.736720 9479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:45.738775 9479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:45.739705 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:45.739952 9479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:45.740048 9479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "d6c110a2424844b7a50e3a9e6531b9cd"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "f0f0b4775ea412d769943caf5d3e3f7b"
server_key_iv: "c2ecda30c216e1085965a2a222f00f3e"
server_key_version: "encryptionkey@0"
I20260926 09:25:45.740549 9479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:45.764379 9479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:45.767222 9479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:45.767405 9479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:45.767738 9479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:45.768364 9535 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 09:25:45.770027 9479 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 09:25:45.770099 9479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 09:25:45.770140 9479 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 09:25:45.771087 9479 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 09:25:45.771137 9479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 09:25:45.771245 9535 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap starting.
I20260926 09:25:45.779356 9479 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44609
I20260926 09:25:45.779783 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44609 every 8 connection(s)
I20260926 09:25:45.779909 9479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:25:45.782492 9535 log.cc:824] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:25:45.783736 9535 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260926 09:25:45.784027 9535 tablet_bootstrap.cc:492] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Bootstrap complete.
I20260926 09:25:45.785059 9535 ts_tablet_manager.cc:1397] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260926 09:25:45.786504 9535 raft_consensus.cc:348] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:45.787241 9535 raft_consensus.cc:729] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c110a2424844b7a50e3a9e6531b9cd, State: Initialized, Role: FOLLOWER
I20260926 09:25:45.787438 9535 consensus_queue.cc:260] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:45.787729 9535 ts_tablet_manager.cc:1428] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 09:25:45.789530 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9479
I20260926 09:25:45.789871 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:36835
--local_ip_for_outbound_sockets=127.24.235.66
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=46801
--webserver_interface=127.24.235.66
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:45.856128 9645 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:25:45.780747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:44009)
0926 09:25:45.780983 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.781063 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:45.781861 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:45.855885 (+ 74024us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
W20260926 09:25:45.856465 9643 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:41015: Client connection negotiation failed: client connection to 127.24.235.126:41015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 09:25:45.856870 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:45.856718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:52403)
0926 09:25:45.856778 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.856835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:25:45.857234 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:45.857133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:36605)
0926 09:25:45.857186 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.857214 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:25:45.857368 9643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 09:25:45.874759 9649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:45.874989 9649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:45.875016 9649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:45.876243 9649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:45.876289 9649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:45.876318 9649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:25:45.877955 9649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:36835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=46801
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:45.878237 9649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:45.878492 9649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:45.883949 9654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:45.883970 9657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:45.883965 9655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:45.884268 9649 server_base.cc:1034] running on GCE node
I20260926 09:25:45.884501 9649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:45.884716 9649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:45.885860 9649 hybrid_clock.cc:648] HybridClock initialized: now 1790414745885849 us; error 31 us; skew 500 ppm
Sep 26 09:25:45 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414745, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:45.893122 9649 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:25:45.893651 9649 webserver.cc:460] Webserver started at http://127.24.235.66:46801/ using document root <none> and password file <none>
I20260926 09:25:45.893918 9649 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:45.893966 9649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:45.895519 9649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:45.896308 9664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:45.896572 9649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:45.896710 9649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "365d262b78c01f4139340761fd1da777"
server_key_iv: "f5e9babee6d4b01bd8066d94969c938a"
server_key_version: "encryptionkey@0"
I20260926 09:25:45.897166 9649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:45.917455 9649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:45.920176 9649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:45.920370 9649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:45.920662 9649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:45.921075 9649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:45.921139 9649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:45.921185 9649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:45.921216 9649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:45.928360 9649 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:36835
I20260926 09:25:45.928406 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:36835 every 8 connection(s)
I20260926 09:25:45.928802 9649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:25:45.930030 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:45.929549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:56315)
0926 09:25:45.929788 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.929870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260926 09:25:45.930471 9778 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:41015: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
I20260926 09:25:45.930851 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:45.930715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:40575)
0926 09:25:45.930794 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.930827 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 09:25:45.931236 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:45.931141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:49733)
0926 09:25:45.931193 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:45.931218 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 09:25:45.931375 9778 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:25:45.936210 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9649
I20260926 09:25:45.936522 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:34459
--local_ip_for_outbound_sockets=127.24.235.67
--tserver_master_addrs=127.24.235.126:41015
--webserver_port=38797
--webserver_interface=127.24.235.67
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:25:46.021236 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:25:46.021554 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:25:46.021596 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:25:46.022833 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:25:46.022886 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:25:46.022913 9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:25:46.024456 9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:34459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=38797
--tserver_master_addrs=127.24.235.126:41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:25:46.024727 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:25:46.024972 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:25:46.030709 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:25:46.030647 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:46.030930 9781 server_base.cc:1034] running on GCE node
W20260926 09:25:46.030722 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:25:46.031292 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:25:46.031589 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:25:46.032770 9781 hybrid_clock.cc:648] HybridClock initialized: now 1790414746032749 us; error 42 us; skew 500 ppm
Sep 26 09:25:46 dist-test-slave-7zlw krb5kdc[8337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414746, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:25:46.035605 9781 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:25:46.036125 9781 webserver.cc:460] Webserver started at http://127.24.235.67:38797/ using document root <none> and password file <none>
I20260926 09:25:46.036361 9781 fs_manager.cc:359] Metadata directory not provided
I20260926 09:25:46.036432 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:25:46.037941 9781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:25:46.038548 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:46.038729 9781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:25:46.038837 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db"
format_stamp: "Formatted at 2026-09-26 09:25:04 on dist-test-slave-7zlw"
server_key: "e5b064c4efadbfdd7574f516402224ec"
server_key_iv: "e61b0361dfab786c81d057d29273bf74"
server_key_version: "encryptionkey@0"
I20260926 09:25:46.039203 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:25:46.056820 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:25:46.059286 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:25:46.059463 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:25:46.059722 9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:25:46.060057 9781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:25:46.060111 9781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:46.060148 9781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:25:46.060195 9781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:25:46.066622 9781 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:34459
I20260926 09:25:46.066685 9909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:34459 every 8 connection(s)
I20260926 09:25:46.067015 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:25:46.068078 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:46.067676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:35515)
0926 09:25:46.067841 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:46.067920 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
W20260926 09:25:46.068574 9910 heartbeater.cc:644] Failed to heartbeat to 127.24.235.126:41015 (0 consecutive failures): Network error: Failed to ping master at 127.24.235.126:41015: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
I20260926 09:25:46.068902 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:46.068808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:53601)
0926 09:25:46.068856 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:46.068884 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:25:46.069216 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:46.069160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:38455)
0926 09:25:46.069182 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:46.069202 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 09:25:46.069340 9910 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 09:25:46.072355 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9781
I20260926 09:25:46.858055 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:46.857811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:43335)
0926 09:25:46.857956 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:46.857994 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260926 09:25:46.932056 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:46.931814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:51055)
0926 09:25:46.931939 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:46.931984 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260926 09:25:47.061856 9916 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:47.062013 9916 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:47.062399 9916 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
Sep 26 09:25:47 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414745, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.66@KRBTEST.COM
Sep 26 09:25:47 dist-test-slave-7zlw krb5kdc[8337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414745, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.67@KRBTEST.COM
I20260926 09:25:47.069963 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.069705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:48853)
0926 09:25:47.069860 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:47.069914 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260926 09:25:47.074187 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.062740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:56519 (local address 127.24.235.67:34459)
0926 09:25:47.062915 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:47.062927 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:47.062956 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:25:47.063386 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:47.063388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:47.063436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:47.063585 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:47.064297 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.065278 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.066224 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.066730 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.067188 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:47.070283 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:47.070287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:47.070287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:47.070298 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:47.072067 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072639 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:47.072643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:47.072643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:47.072782 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:47.073132 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:47.073133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:47.073134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:47.073468 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:47.073637 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:47.073937 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 09:25:47.074077 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260926 09:25:47.074188 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.062727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:45659 (local address 127.24.235.66:36835)
0926 09:25:47.062867 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 09:25:47.062878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:25:47.062919 (+ 41us) server_negotiation.cc:407] Connection header received
0926 09:25:47.063378 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:25:47.063381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:25:47.063423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:25:47.063566 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:25:47.064275 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.065220 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.066225 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.066690 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.066826 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:47.069913 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:25:47.069917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:25:47.069917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:25:47.069930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:25:47.071951 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072639 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:47.072643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:47.072643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:47.072784 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:25:47.073132 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:25:47.073133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:25:47.073134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:25:47.073468 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:25:47.073566 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 09:25:47.073937 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 09:25:47.074076 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260926 09:25:47.074239 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.062572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:34459 (local address 127.24.235.65:56519)
0926 09:25:47.062641 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:47.062654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:47.062724 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:47.063617 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:47.063620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:47.063970 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:47.064154 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:47.064159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.065386 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.065387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:47.066122 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:47.066124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.066833 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.066834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:47.066903 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:47.067515 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:47.067526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:47.070178 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:47.072141 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:47.072142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:47.072518 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:47.072873 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:47.072874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:47.073064 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:47.073683 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:47.073684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:47.073733 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:25:47.073870 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 09:25:47.074142 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":162}
I20260926 09:25:47.074241 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.062572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:36835 (local address 127.24.235.65:45659)
0926 09:25:47.062795 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:47.062805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:25:47.062863 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:25:47.063618 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:25:47.063621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:25:47.063971 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:25:47.064154 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:47.064160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.065316 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.065317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:47.066120 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:25:47.066122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:25:47.066793 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:25:47.066794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:25:47.066902 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:25:47.067568 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:25:47.067573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:25:47.069679 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:25:47.072039 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:47.072047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:47.072518 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:47.072873 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:25:47.072874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:25:47.072874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:25:47.073064 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:25:47.073557 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:25:47.073563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:25:47.073733 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 09:25:47.073870 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 09:25:47.074085 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":150,"thread_start_us":46,"threads_started":1}
W20260926 09:25:47.075101 9530 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:25:47.075727 9531 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:47.075773 9531 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6c110a2424844b7a50e3a9e6531b9cd; no voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e
I20260926 09:25:47.075883 9916 raft_consensus.cc:2726] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:25:47.858871 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.858628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:46057)
0926 09:25:47.858778 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:47.858821 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260926 09:25:47.932785 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:47.932563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:55969)
0926 09:25:47.932707 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:47.932742 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260926 09:25:48.070608 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:48.070368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:36295)
0926 09:25:48.070509 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:48.070556 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260926 09:25:48.859602 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:48.859400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:33793)
0926 09:25:48.859520 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:48.859554 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 09:25:48.933507 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:48.933275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:42241)
0926 09:25:48.933427 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:48.933462 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260926 09:25:49.071308 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:49.071106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:59191)
0926 09:25:49.071229 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:49.071265 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 09:25:49.213919 9927 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:49.214066 9927 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:49.214290 9927 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
W20260926 09:25:49.214922 9531 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:25:49.215075 9530 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:49.215114 9530 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6c110a2424844b7a50e3a9e6531b9cd; no voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e
I20260926 09:25:49.215215 9927 raft_consensus.cc:2726] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:25:49.860507 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:49.860269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:36423)
0926 09:25:49.860424 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:49.860459 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260926 09:25:49.934288 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:49.934042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:47997)
0926 09:25:49.934195 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:49.934241 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260926 09:25:50.072126 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:50.071857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:40637)
0926 09:25:50.071997 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:50.072054 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 09:25:50.861357 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:50.861134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:39085)
0926 09:25:50.861271 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:50.861311 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260926 09:25:50.934933 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:50.934724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:44609)
0926 09:25:50.934842 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:50.934876 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260926 09:25:51.072834 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:51.072619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:44237)
0926 09:25:51.072745 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:51.072786 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260926 09:25:51.103287 9934 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:51.103384 9934 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:51.103611 9934 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
W20260926 09:25:51.104102 9530 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:25:51.104172 9531 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:51.104202 9530 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6c110a2424844b7a50e3a9e6531b9cd; no voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e
I20260926 09:25:51.104286 9934 raft_consensus.cc:2726] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:25:51.862190 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:51.861921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:42995)
0926 09:25:51.862087 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:51.862139 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260926 09:25:51.935686 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:51.935427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:52519)
0926 09:25:51.935589 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:51.935637 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260926 09:25:52.073525 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:52.073304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:45051)
0926 09:25:52.073439 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:52.073480 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 09:25:52.862912 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:52.862715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:43929)
0926 09:25:52.862831 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:52.862865 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260926 09:25:52.936617 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:52.936297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:45433)
0926 09:25:52.936471 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:52.936569 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260926 09:25:53.074299 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:53.074057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:47657)
0926 09:25:53.074219 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:53.074250 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260926 09:25:53.863759 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:53.863497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:35449)
0926 09:25:53.863651 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:53.863701 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260926 09:25:53.937487 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:53.937268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:52879)
0926 09:25:53.937398 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:53.937439 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260926 09:25:54.074990 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:54.074787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:37821)
0926 09:25:54.074907 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:54.074945 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 09:25:54.864550 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:54.864335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:50039)
0926 09:25:54.864466 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:54.864501 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260926 09:25:54.938217 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:54.937970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:37711)
0926 09:25:54.938111 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:54.938168 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 09:25:55.075829 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:55.075534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:56965)
0926 09:25:55.075728 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:55.075777 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260926 09:25:55.489398 9947 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:25:55.489504 9947 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:25:55.489708 9947 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
W20260926 09:25:55.490125 9530 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:25:55.490268 9531 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:25:55.490339 9531 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6c110a2424844b7a50e3a9e6531b9cd; no voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e
I20260926 09:25:55.490427 9947 raft_consensus.cc:2726] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:25:55.865427 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:55.865184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:43627)
0926 09:25:55.865322 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:55.865365 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260926 09:25:55.938962 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:55.938759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:45453)
0926 09:25:55.938880 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:55.938916 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260926 09:25:56.076642 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:56.076374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:57281)
0926 09:25:56.076509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:56.076593 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260926 09:25:56.866303 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:56.866032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:43483)
0926 09:25:56.866197 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:56.866248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260926 09:25:56.939755 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:56.939459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:44659)
0926 09:25:56.939656 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:56.939698 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260926 09:25:57.077369 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:57.077158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:42115)
0926 09:25:57.077283 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:57.077323 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 09:25:57.867044 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:57.866816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:50631)
0926 09:25:57.866961 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:57.866997 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260926 09:25:57.940418 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:57.940215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:56403)
0926 09:25:57.940338 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:57.940373 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 09:25:58.078063 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:58.077838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:44939)
0926 09:25:58.077979 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:58.078015 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260926 09:25:58.867846 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:58.867616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:57859)
0926 09:25:58.867749 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:58.867794 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260926 09:25:58.941210 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:58.940956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:36129)
0926 09:25:58.941121 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:58.941165 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":126,"threads_started":1}
I20260926 09:25:59.079000 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:59.078741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:48999)
0926 09:25:59.078908 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:59.078952 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260926 09:25:59.868597 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:59.868358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:60839)
0926 09:25:59.868509 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:59.868549 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260926 09:25:59.942046 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:25:59.941737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:47001)
0926 09:25:59.941907 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 09:25:59.941954 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260926 09:26:00.079716 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:00.079480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:35577)
0926 09:26:00.079628 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:00.079667 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260926 09:26:00.824558 9964 raft_consensus.cc:482] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:00.824671 9964 raft_consensus.cc:504] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a409cd42ddc4ae8a7e9f68ad55b78db" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 34459 } } peers { permanent_uuid: "cf9423d1ffdd4ae89ba67d35bc188b2e" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 36835 } } peers { permanent_uuid: "d6c110a2424844b7a50e3a9e6531b9cd" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44609 } }
I20260926 09:26:00.824880 9964 leader_election.cc:288] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459), cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835)
W20260926 09:26:00.825428 9531 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a409cd42ddc4ae8a7e9f68ad55b78db (127.24.235.67:34459): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
W20260926 09:26:00.825522 9530 leader_election.cc:341] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf9423d1ffdd4ae89ba67d35bc188b2e (127.24.235.66:36835): Not found: Tablet not found: 79037f1e7d9741ab9c20cf5bf3b13321
I20260926 09:26:00.825632 9530 leader_election.cc:302] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6c110a2424844b7a50e3a9e6531b9cd; no voters: 0a409cd42ddc4ae8a7e9f68ad55b78db, cf9423d1ffdd4ae89ba67d35bc188b2e
I20260926 09:26:00.825716 9964 raft_consensus.cc:2726] T 79037f1e7d9741ab9c20cf5bf3b13321 P d6c110a2424844b7a50e3a9e6531b9cd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 09:26:00.869611 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:00.869330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.65:50019)
0926 09:26:00.869474 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:00.869518 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260926 09:26:00.942795 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:00.942534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.66:44873)
0926 09:26:00.942688 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:00.942734 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260926 09:26:01.075891 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9479
I20260926 09:26:01.080571 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:01.080334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:41015 (local address 127.24.235.67:42541)
0926 09:26:01.080481 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:01.080524 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.126:41015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260926 09:26:01.081761 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9649
I20260926 09:26:01.086716 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9781
2026-09-26T09:26:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58487 ms)
[----------] 30 tests from SecurityITest (238821 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:01 dist-test-slave-7zlw krb5kdc[9970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:01 dist-test-slave-7zlw krb5kdc[9970](info): set up 2 sockets
Sep 26 09:26:01 dist-test-slave-7zlw krb5kdc[9970](info): commencing operation
krb5kdc: starting...
W20260926 09:26:02.180929 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:02.309156 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45743
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:02.394994 9983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:02.395228 9983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:02.395253 9983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:02.396479 9983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:02.396536 9983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:02.396553 9983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:02.396569 9983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:02.396584 9983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:02.398197 9983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:02.398469 9983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:02.398741 9983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:02.404413 9991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.404415 9989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.404413 9988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.404672 9983 server_base.cc:1034] running on GCE node
I20260926 09:26:02.404959 9983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:02.405289 9983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:02.406445 9983 hybrid_clock.cc:648] HybridClock initialized: now 1790414762406419 us; error 41 us; skew 500 ppm
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:02.409273 9983 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:02.409727 9983 webserver.cc:460] Webserver started at http://127.24.235.126:35849/ using document root <none> and password file <none>
I20260926 09:26:02.409921 9983 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:02.409961 9983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:02.410058 9983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:02.411114 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "f1033ae5f3604325b44fac80552379d0"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "826c4d78590c99e2f279c3e67ee08854"
server_key_iv: "bbdc84ea9a41935c239933491e027990"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.411417 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "f1033ae5f3604325b44fac80552379d0"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "826c4d78590c99e2f279c3e67ee08854"
server_key_iv: "bbdc84ea9a41935c239933491e027990"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.412680 9983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:02.413472 9998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.413666 9983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:26:02.413756 9983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "f1033ae5f3604325b44fac80552379d0"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "826c4d78590c99e2f279c3e67ee08854"
server_key_iv: "bbdc84ea9a41935c239933491e027990"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.413834 9983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:02.421504 9983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:02.423872 9983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:02.424024 9983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:02.428197 9983 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:45743
I20260926 09:26:02.428260 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:45743 every 8 connection(s)
I20260926 09:26:02.428622 9983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:02.429562 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:02.431921 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: Bootstrap starting.
I20260926 09:26:02.432533 10051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:02.432857 10051 log.cc:824] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:02.433631 10051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: No bootstrap required, opened a new log
I20260926 09:26:02.434751 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 9983
I20260926 09:26:02.434841 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:02.435060 25517 external_mini_cluster.cc:1428] Setting key a84667527326b3c8d853e9cc54caa27e
I20260926 09:26:02.435559 10051 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } }
I20260926 09:26:02.435783 10051 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:02.435854 10051 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1033ae5f3604325b44fac80552379d0, State: Initialized, Role: FOLLOWER
I20260926 09:26:02.436000 10051 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } }
I20260926 09:26:02.436084 10051 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:02.436112 10051 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:02.436153 10051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:02.437258 10051 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } }
I20260926 09:26:02.437371 10051 leader_election.cc:302] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1033ae5f3604325b44fac80552379d0; no voters:
I20260926 09:26:02.437551 10051 leader_election.cc:288] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:02.437628 10056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:02.437804 10056 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [term 1 LEADER]: Becoming Leader. State: Replica: f1033ae5f3604325b44fac80552379d0, State: Running, Role: LEADER
I20260926 09:26:02.437812 10051 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:02.438582 10056 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } }
I20260926 09:26:02.439400 10057 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1033ae5f3604325b44fac80552379d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } } }
I20260926 09:26:02.439402 10058 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1033ae5f3604325b44fac80552379d0. Latest consensus state: current_term: 1 leader_uuid: "f1033ae5f3604325b44fac80552379d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1033ae5f3604325b44fac80552379d0" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45743 } } }
I20260926 09:26:02.439538 10057 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:02.439544 10058 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:02.439853 10071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:02.440440 10071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:02.442277 10071 catalog_manager.cc:1347] Generated new cluster ID: f6c0e9f9bda748288083f9ee716ee2b9
I20260926 09:26:02.442330 10071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:02.448215 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.435682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.24.235.126:45743)
0926 09:26:02.435947 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:02.435958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:02.435991 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:26:02.436110 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:02.436118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:02.436176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:02.436362 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:02.436959 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.438205 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.439418 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.440053 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.440166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.443273 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:02.443277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:02.443277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:02.443289 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:02.445353 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.445717 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.445719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.445720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.446434 (+ 714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.446641 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.446642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.446642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.446944 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:02.447193 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:02.447830 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 09:26:02.447977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260926 09:26:02.460395 10071 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:02.461202 10071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:02.464571 10071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: Generated new TSK 0
I20260926 09:26:02.464763 10071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:26:02.496502 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:02.579653 10077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:02.579912 10077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:02.579949 10077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:02.581144 10077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:02.581220 10077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:02.581266 10077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:02.582779 10077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:02.583043 10077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:02.583281 10077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:02.588832 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.588838 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.588941 10077 server_base.cc:1034] running on GCE node
W20260926 09:26:02.589313 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.589586 10077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:02.589998 10077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:02.591202 10077 hybrid_clock.cc:648] HybridClock initialized: now 1790414762591193 us; error 47 us; skew 500 ppm
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:02.594007 10077 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:26:02.594568 10077 webserver.cc:460] Webserver started at http://127.24.235.65:43985/ using document root <none> and password file <none>
I20260926 09:26:02.594828 10077 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:02.594877 10077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:02.595062 10077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:02.596191 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "cbc1431fe26945c1982f9281d9269eeb"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "60c70633005df2a31c5273514e09b7a9"
server_key_iv: "b75138e85b764e840fd4225575818914"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.596580 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "cbc1431fe26945c1982f9281d9269eeb"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "60c70633005df2a31c5273514e09b7a9"
server_key_iv: "b75138e85b764e840fd4225575818914"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.598080 10077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260926 09:26:02.598939 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.599121 10077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:02.599220 10077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "cbc1431fe26945c1982f9281d9269eeb"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "60c70633005df2a31c5273514e09b7a9"
server_key_iv: "b75138e85b764e840fd4225575818914"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.599303 10077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:02.621917 10077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:02.624251 10077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:02.624442 10077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:02.624749 10077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:02.625162 10077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:02.625202 10077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.625227 10077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:02.625279 10077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.632100 10077 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:38039
I20260926 09:26:02.632161 10205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:38039 every 8 connection(s)
I20260926 09:26:02.632544 10077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:02.642520 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10077
I20260926 09:26:02.642554 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.633461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:53381 (local address 127.24.235.126:45743)
0926 09:26:02.633521 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:02.633524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:02.633691 (+ 167us) server_negotiation.cc:407] Connection header received
0926 09:26:02.634272 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:02.634274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:02.634292 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:02.634350 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:02.635416 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.635953 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.636990 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.637234 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.637326 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.639760 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:02.639761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:02.639762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:02.639770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:02.641005 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.641476 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.641477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.641478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.641602 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.641875 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.641876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.641876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.641998 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:02.642078 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:02.642360 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 09:26:02.642450 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 09:26:02.642645 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:02.642737 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.633368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45743 (local address 127.24.235.65:53381)
0926 09:26:02.633543 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:02.633592 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:02.634159 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:02.634427 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:02.634431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:02.635090 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:02.635311 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.635316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.636056 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.636057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.636876 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.636895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.637340 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.637341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.637477 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.638029 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:02.638038 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:02.639656 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:02.641138 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.641145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.641147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.641383 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.641681 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.641681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.641682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.641804 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.642068 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:02.642075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:02.642220 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 09:26:02.642326 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 09:26:02.642491 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260926 09:26:02.642859 25517 external_mini_cluster.cc:1428] Setting key 4aed2c192a77d8893678597b64239d83
I20260926 09:26:02.643318 10206 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45743
I20260926 09:26:02.643415 10206 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:02.643610 10206 heartbeater.cc:511] Master 127.24.235.126:45743 requested a full tablet report, sending...
I20260926 09:26:02.644214 10014 ts_manager.cc:194] Registered new tserver with Master: cbc1431fe26945c1982f9281d9269eeb (127.24.235.65:38039)
I20260926 09:26:02.645165 10014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:53381
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:26:02.668983 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:02.753258 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:02.753516 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:02.753562 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:02.754719 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:02.754756 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:02.754783 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:02.756147 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:02.756337 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:02.756511 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:02.761736 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.761709 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.761840 10211 server_base.cc:1034] running on GCE node
W20260926 09:26:02.761710 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.762143 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:02.762347 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:02.763492 10211 hybrid_clock.cc:648] HybridClock initialized: now 1790414762763480 us; error 29 us; skew 500 ppm
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:02.766197 10211 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:26:02.766661 10211 webserver.cc:460] Webserver started at http://127.24.235.66:33577/ using document root <none> and password file <none>
I20260926 09:26:02.766889 10211 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:02.766958 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:02.767104 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:02.768162 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "0e6b94567b0b402bb4189e482c7a80b9"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "a7b42b25c173e140c45092016e4f07ed"
server_key_iv: "b11119723c0ca502beb4c878a848999e"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.768544 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "0e6b94567b0b402bb4189e482c7a80b9"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "a7b42b25c173e140c45092016e4f07ed"
server_key_iv: "b11119723c0ca502beb4c878a848999e"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.769955 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:02.770716 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.770905 10211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:02.771008 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "0e6b94567b0b402bb4189e482c7a80b9"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "a7b42b25c173e140c45092016e4f07ed"
server_key_iv: "b11119723c0ca502beb4c878a848999e"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.771080 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:02.788774 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:02.791203 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:02.791381 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:02.791643 10211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:02.791965 10211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:02.792017 10211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.792060 10211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:02.792099 10211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.798449 10211 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:34423
I20260926 09:26:02.798513 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:34423 every 8 connection(s)
I20260926 09:26:02.798841 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:02.804540 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10211
I20260926 09:26:02.804646 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:02.804883 25517 external_mini_cluster.cc:1428] Setting key 8d9e010feb59cb6aee7ab82b44652dc7
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:02.809520 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.799569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45743 (local address 127.24.235.66:40407)
0926 09:26:02.799733 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:02.799783 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:02.800280 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:02.800537 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:02.800540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:02.801167 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:02.801376 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.801381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.802126 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.802127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.803051 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.803054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.803469 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.803471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.803613 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.804106 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:02.804114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:02.806084 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:02.807875 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.807883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.807885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.808225 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.808470 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.808470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.808471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.808610 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.808944 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:02.808950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:02.809128 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 09:26:02.809225 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 09:26:02.809352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260926 09:26:02.809512 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.799600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:40407 (local address 127.24.235.126:45743)
0926 09:26:02.799665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:02.799667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:02.799884 (+ 217us) server_negotiation.cc:407] Connection header received
0926 09:26:02.800380 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:02.800381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:02.800397 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:02.800484 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:02.801470 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.802010 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.803139 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.803341 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.803460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.806192 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:02.806193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:02.806194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:02.806202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:02.807730 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.808332 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.808334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.808334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.808391 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.808711 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.808712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.808712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.808866 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:02.808942 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:02.809271 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 09:26:02.809371 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:26:02.810292 10340 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45743
I20260926 09:26:02.810410 10340 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:02.810586 10340 heartbeater.cc:511] Master 127.24.235.126:45743 requested a full tablet report, sending...
I20260926 09:26:02.811102 10014 ts_manager.cc:194] Registered new tserver with Master: 0e6b94567b0b402bb4189e482c7a80b9 (127.24.235.66:34423)
I20260926 09:26:02.811703 10014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:40407
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:26:02.832281 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:02.916291 10345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:02.916510 10345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:02.916535 10345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:02.917850 10345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:02.917892 10345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:02.917919 10345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:02.919401 10345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:45743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:02.919655 10345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:02.919895 10345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:02.925067 10350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.925087 10353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:02.925047 10351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:02.925380 10345 server_base.cc:1034] running on GCE node
I20260926 09:26:02.925673 10345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:02.925942 10345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:02.927134 10345 hybrid_clock.cc:648] HybridClock initialized: now 1790414762927113 us; error 33 us; skew 500 ppm
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:02.929751 10345 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:26:02.930249 10345 webserver.cc:460] Webserver started at http://127.24.235.67:42533/ using document root <none> and password file <none>
I20260926 09:26:02.930475 10345 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:02.930543 10345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:02.930701 10345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:02.931806 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "d942c4efef6149b889993a497586dac7"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "58bedcf4e7bd4f003f81172952496658"
server_key_iv: "1c7aa0973e874d65203f2a813ff6714a"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.932184 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "d942c4efef6149b889993a497586dac7"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "58bedcf4e7bd4f003f81172952496658"
server_key_iv: "1c7aa0973e874d65203f2a813ff6714a"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.933759 10345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:02.934641 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.934875 10345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:02.934976 10345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "d942c4efef6149b889993a497586dac7"
format_stamp: "Formatted at 2026-09-26 09:26:02 on dist-test-slave-7zlw"
server_key: "58bedcf4e7bd4f003f81172952496658"
server_key_iv: "1c7aa0973e874d65203f2a813ff6714a"
server_key_version: "encryptionkey@0"
I20260926 09:26:02.935068 10345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:02.964031 10345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:02.966295 10345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:02.966460 10345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:02.966707 10345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:02.967036 10345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:02.967090 10345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.967160 10345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:02.967192 10345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:02.973301 10345 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:46269
I20260926 09:26:02.973428 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:46269 every 8 connection(s)
I20260926 09:26:02.973824 10345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:02.978163 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10345
I20260926 09:26:02.978293 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:02.978492 25517 external_mini_cluster.cc:1428] Setting key 7294f6decd97652a15ab3d0378634c72
Sep 26 09:26:02 dist-test-slave-7zlw krb5kdc[9970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414762, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:02.983188 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.974572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:49957 (local address 127.24.235.126:45743)
0926 09:26:02.974615 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:02.974618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:02.974888 (+ 270us) server_negotiation.cc:407] Connection header received
0926 09:26:02.975436 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:02.975438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:02.975452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:02.975503 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:02.976517 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.977090 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.978065 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.978279 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.978362 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.980728 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:02.980729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:02.980730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:02.980736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:02.982013 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.982394 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.982394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.982395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.982441 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.982664 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.982664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.982665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.982769 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:02.982818 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:02.983058 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 09:26:02.983122 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:26:02.983371 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.974604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45743 (local address 127.24.235.67:49957)
0926 09:26:02.974748 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:02.974795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:02.975321 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:02.975615 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:02.975619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:02.976221 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:02.976415 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.976420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.977172 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.977172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.977972 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:02.977973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.978399 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.978400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:02.978568 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.979192 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:02.979201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:02.980647 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:02.982087 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.982094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.982096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.982326 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.982500 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:02.982500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:02.982501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:02.982597 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:02.982826 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:02.982830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:02.982947 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 09:26:02.983029 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:26:02.983148 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260926 09:26:02.983824 10474 heartbeater.cc:347] Connected to a master server at 127.24.235.126:45743
I20260926 09:26:02.983903 10474 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:02.984100 10474 heartbeater.cc:511] Master 127.24.235.126:45743 requested a full tablet report, sending...
I20260926 09:26:02.984479 10014 ts_manager.cc:194] Registered new tserver with Master: d942c4efef6149b889993a497586dac7 (127.24.235.67:46269)
I20260926 09:26:02.984882 10014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:49957
I20260926 09:26:02.990129 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:02.997790 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:02.991521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.24.235.126:45743)
0926 09:26:02.991616 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:02.991618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:02.991699 (+ 81us) server_negotiation.cc:407] Connection header received
0926 09:26:02.991766 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:02.991767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:02.991777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:02.991847 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:02.992396 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.992897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.993702 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:02.993894 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:02.993997 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:02.995151 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:02.995153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:02.995153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:02.995161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:02.996288 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.996673 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.996675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.996675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.996757 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:02.997054 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:02.997055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:02.997055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:02.997290 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:02.997410 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:02.997573 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 09:26:02.997679 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:26:02.999927 10014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:03.002490 10014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:03.010318 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.006010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38039 (local address 127.0.0.1:51800)
0926 09:26:03.006618 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.006626 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.006703 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.007006 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.007007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.007009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.007058 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.007060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.008296 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.008297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.009277 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.009280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.009959 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.009960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.010039 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010053 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.010169 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.010227 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":102,"threads_started":1}
I20260926 09:26:03.010318 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.006247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.24.235.65:38039)
0926 09:26:03.006458 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.006462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.006676 (+ 214us) server_negotiation.cc:407] Connection header received
0926 09:26:03.006815 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.006818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.006836 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.006892 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.007141 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.008185 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.009360 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.009861 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.009947 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010009 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.010186 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.010227 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260926 09:26:03.010705 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.005989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:46269 (local address 127.0.0.1:57506)
0926 09:26:03.006502 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.006512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.006549 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.006826 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.006827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.006829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.006872 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.006873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.007850 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.007851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.009310 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.009312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.010320 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.010321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.010392 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010400 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.010450 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.010508 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":13,"thread_start_us":59,"threads_started":1}
I20260926 09:26:03.010728 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.006404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.24.235.67:46269)
0926 09:26:03.006609 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.006613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.006623 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:03.006661 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.006662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.006677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.006754 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.006939 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.007735 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.009414 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.010137 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.010210 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010279 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.010595 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.010646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260926 09:26:03.010828 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.006200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.24.235.66:34423)
0926 09:26:03.006404 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.006410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.006425 (+ 15us) server_negotiation.cc:407] Connection header received
0926 09:26:03.006479 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.006481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.006497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.006803 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.006876 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.007965 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.009414 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.010137 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.010236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010296 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.010679 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.010728 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260926 09:26:03.010612 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.005989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:34423 (local address 127.0.0.1:35890)
0926 09:26:03.006281 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.006326 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.006397 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.006629 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.006634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.006637 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.006726 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.006728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.008069 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.008070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.009309 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.009312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.010327 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.010328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.010392 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.010398 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.010450 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.010497 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":36,"spinlock_wait_cycles":1280,"thread_start_us":113,"threads_started":1}
I20260926 09:26:03.011510 10140 tablet_service.cc:1467] Processing CreateTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 (DEFAULT_TABLE table=test-table [id=df9fbcdb98a74ca7a5b8e9f9520220fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.011864 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cbb6122a6140f4b6a2ced60c635b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.013424 10408 tablet_service.cc:1467] Processing CreateTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 (DEFAULT_TABLE table=test-table [id=df9fbcdb98a74ca7a5b8e9f9520220fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.013756 10408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cbb6122a6140f4b6a2ced60c635b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.014635 10491 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: Bootstrap starting.
I20260926 09:26:03.015218 10491 tablet_bootstrap.cc:654] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.015700 10491 log.cc:824] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.016517 10491 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: No bootstrap required, opened a new log
I20260926 09:26:03.016508 10493 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Bootstrap starting.
I20260926 09:26:03.016590 10491 ts_tablet_manager.cc:1397] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:26:03.017156 10493 tablet_bootstrap.cc:654] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.017135 10274 tablet_service.cc:1467] Processing CreateTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 (DEFAULT_TABLE table=test-table [id=df9fbcdb98a74ca7a5b8e9f9520220fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.017463 10274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49cbb6122a6140f4b6a2ced60c635b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.017752 10493 log.cc:824] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.018466 10493 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: No bootstrap required, opened a new log
I20260926 09:26:03.018548 10493 ts_tablet_manager.cc:1397] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:26:03.018725 10491 raft_consensus.cc:348] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.018930 10491 raft_consensus.cc:374] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.018985 10491 raft_consensus.cc:729] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc1431fe26945c1982f9281d9269eeb, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.019096 10491 consensus_queue.cc:260] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.019389 10491 ts_tablet_manager.cc:1428] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:03.019449 10206 heartbeater.cc:503] Master 127.24.235.126:45743 was elected leader, sending a full tablet report...
I20260926 09:26:03.020298 10493 raft_consensus.cc:348] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.020452 10493 raft_consensus.cc:374] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.020498 10493 raft_consensus.cc:729] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d942c4efef6149b889993a497586dac7, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.020623 10493 consensus_queue.cc:260] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.020676 10496 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Bootstrap starting.
I20260926 09:26:03.020915 10493 ts_tablet_manager.cc:1428] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 09:26:03.020934 10474 heartbeater.cc:503] Master 127.24.235.126:45743 was elected leader, sending a full tablet report...
I20260926 09:26:03.022008 10496 tablet_bootstrap.cc:654] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.022454 10496 log.cc:824] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.023065 10496 tablet_bootstrap.cc:492] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: No bootstrap required, opened a new log
I20260926 09:26:03.023118 10496 ts_tablet_manager.cc:1397] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 09:26:03.024676 10496 raft_consensus.cc:348] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.024816 10496 raft_consensus.cc:374] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.024853 10496 raft_consensus.cc:729] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6b94567b0b402bb4189e482c7a80b9, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.024976 10496 consensus_queue.cc:260] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.025240 10496 ts_tablet_manager.cc:1428] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:26:03.025286 10340 heartbeater.cc:503] Master 127.24.235.126:45743 was elected leader, sending a full tablet report...
W20260926 09:26:03.049401 10341 tablet.cc:2369] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:03.059365 10499 raft_consensus.cc:482] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:03.059495 10499 raft_consensus.cc:504] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.059939 10499 leader_election.cc:288] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d942c4efef6149b889993a497586dac7 (127.24.235.67:46269), cbc1431fe26945c1982f9281d9269eeb (127.24.235.65:38039)
I20260926 09:26:03.063859 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.060223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:34525 (local address 127.24.235.67:46269)
0926 09:26:03.060281 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.060283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.060319 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:26:03.060351 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.060351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.060359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.060395 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.060616 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.061395 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.062911 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.063503 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.063584 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.063592 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.063750 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.063787 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:26:03.063859 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.060225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:46349 (local address 127.24.235.65:38039)
0926 09:26:03.060272 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.060274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.060438 (+ 164us) server_negotiation.cc:407] Connection header received
0926 09:26:03.060469 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.060470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.060485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.060531 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.060727 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.061446 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.062911 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.063503 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.063584 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.063592 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.063750 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.063788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:03.063905 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.060114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:46269 (local address 127.24.235.66:34525)
0926 09:26:03.060205 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.060219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.060275 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.060424 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.060425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.060427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.060509 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.060510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.061503 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.061504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.062813 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.062817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.063593 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.063594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.063675 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.063677 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.063733 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.063778 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":41}
I20260926 09:26:03.064119 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.060155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38039 (local address 127.24.235.66:46349)
0926 09:26:03.060359 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.060367 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.060417 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.060559 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.060560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.060562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.060642 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.060644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.061515 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.061516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.062813 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.062817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.063594 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.063594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.063670 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.063674 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.063733 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.063779 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":51,"thread_start_us":146,"threads_started":1}
I20260926 09:26:03.064373 10428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cbb6122a6140f4b6a2ced60c635b77" candidate_uuid: "0e6b94567b0b402bb4189e482c7a80b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d942c4efef6149b889993a497586dac7" is_pre_election: true
I20260926 09:26:03.064455 10160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cbb6122a6140f4b6a2ced60c635b77" candidate_uuid: "0e6b94567b0b402bb4189e482c7a80b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc1431fe26945c1982f9281d9269eeb" is_pre_election: true
I20260926 09:26:03.064553 10428 raft_consensus.cc:2454] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e6b94567b0b402bb4189e482c7a80b9 in term 0.
I20260926 09:26:03.064580 10160 raft_consensus.cc:2454] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e6b94567b0b402bb4189e482c7a80b9 in term 0.
I20260926 09:26:03.064836 10227 leader_election.cc:302] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e6b94567b0b402bb4189e482c7a80b9, cbc1431fe26945c1982f9281d9269eeb; no voters:
I20260926 09:26:03.064975 10499 raft_consensus.cc:2781] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:03.065129 10499 raft_consensus.cc:482] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:03.065152 10499 raft_consensus.cc:3037] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.065902 10499 raft_consensus.cc:504] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.066045 10499 leader_election.cc:288] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [CANDIDATE]: Term 1 election: Requested vote from peers d942c4efef6149b889993a497586dac7 (127.24.235.67:46269), cbc1431fe26945c1982f9281d9269eeb (127.24.235.65:38039)
I20260926 09:26:03.066197 10160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cbb6122a6140f4b6a2ced60c635b77" candidate_uuid: "0e6b94567b0b402bb4189e482c7a80b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc1431fe26945c1982f9281d9269eeb"
I20260926 09:26:03.066197 10428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49cbb6122a6140f4b6a2ced60c635b77" candidate_uuid: "0e6b94567b0b402bb4189e482c7a80b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d942c4efef6149b889993a497586dac7"
I20260926 09:26:03.066274 10160 raft_consensus.cc:3037] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.066275 10428 raft_consensus.cc:3037] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.067175 10428 raft_consensus.cc:2454] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e6b94567b0b402bb4189e482c7a80b9 in term 1.
I20260926 09:26:03.067176 10160 raft_consensus.cc:2454] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e6b94567b0b402bb4189e482c7a80b9 in term 1.
I20260926 09:26:03.067340 10227 leader_election.cc:302] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e6b94567b0b402bb4189e482c7a80b9, cbc1431fe26945c1982f9281d9269eeb; no voters:
I20260926 09:26:03.067464 10499 raft_consensus.cc:2781] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:03.067636 10499 raft_consensus.cc:686] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e6b94567b0b402bb4189e482c7a80b9, State: Running, Role: LEADER
I20260926 09:26:03.067751 10499 consensus_queue.cc:237] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } }
I20260926 09:26:03.068670 10015 catalog_manager.cc:5495] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e6b94567b0b402bb4189e482c7a80b9 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "0e6b94567b0b402bb4189e482c7a80b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e6b94567b0b402bb4189e482c7a80b9" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 34423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 } health_report { overall_health: UNKNOWN } } }
I20260926 09:26:03.078298 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.075756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.24.235.66:34423)
0926 09:26:03.075821 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.075824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.075833 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:03.075926 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.075927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.075944 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.076013 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:03.076155 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.076636 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.077473 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.077682 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.077755 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.077939 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:03.078048 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:03.078137 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.078182 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.078204 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:03.080030 10428 raft_consensus.cc:1261] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 1 FOLLOWER]: Refusing update from remote peer 0e6b94567b0b402bb4189e482c7a80b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:03.080036 10160 raft_consensus.cc:1261] T 49cbb6122a6140f4b6a2ced60c635b77 P cbc1431fe26945c1982f9281d9269eeb [term 1 FOLLOWER]: Refusing update from remote peer 0e6b94567b0b402bb4189e482c7a80b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:03.080371 10499 consensus_queue.cc:1035] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc1431fe26945c1982f9281d9269eeb" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:03.080513 10501 consensus_queue.cc:1035] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d942c4efef6149b889993a497586dac7" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 46269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:03.083933 10505 mvcc.cc:205] Tried to move back new op lower bound from 7333538869573189632 to 7333538869526110208. Current Snapshot: MvccSnapshot[applied={T|T < 7333538869573189632}]
I20260926 09:26:03.084309 10507 mvcc.cc:205] Tried to move back new op lower bound from 7333538869573189632 to 7333538869526110208. Current Snapshot: MvccSnapshot[applied={T|T < 7333538869573189632}]
I20260926 09:26:03.086680 10014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:03.086790 10014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:03.087576 10014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1033ae5f3604325b44fac80552379d0: Sending DeleteTablet for 3 replicas of tablet 49cbb6122a6140f4b6a2ced60c635b77
I20260926 09:26:03.087996 10274 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:35890
I20260926 09:26:03.088008 10408 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:57506
I20260926 09:26:03.088136 10140 tablet_service.cc:1514] Processing DeleteTablet for tablet 49cbb6122a6140f4b6a2ced60c635b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:51800
I20260926 09:26:03.088348 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10077
I20260926 09:26:03.088388 10514 tablet_replica.cc:330] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: stopping tablet replica
I20260926 09:26:03.089164 10514 raft_consensus.cc:2229] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:03.089308 10514 raft_consensus.cc:2258] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:03.089800 10514 ts_tablet_manager.cc:1905] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:03.091831 10514 ts_tablet_manager.cc:1918] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:03.091910 10514 log.cc:1196] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/49cbb6122a6140f4b6a2ced60c635b77
I20260926 09:26:03.092202 10514 ts_tablet_manager.cc:1939] T 49cbb6122a6140f4b6a2ced60c635b77 P 0e6b94567b0b402bb4189e482c7a80b9: Deleting consensus metadata
I20260926 09:26:03.092636 10000 catalog_manager.cc:4841] TS 0e6b94567b0b402bb4189e482c7a80b9 (127.24.235.66:34423): tablet 49cbb6122a6140f4b6a2ced60c635b77 (table test-table [id=df9fbcdb98a74ca7a5b8e9f9520220fa]) successfully deleted
I20260926 09:26:03.093160 10515 tablet_replica.cc:330] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: stopping tablet replica
I20260926 09:26:03.093309 10515 raft_consensus.cc:2229] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:03.093441 10515 raft_consensus.cc:2258] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:03.093887 10515 ts_tablet_manager.cc:1905] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:03.095934 10515 ts_tablet_manager.cc:1918] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:03.096014 10515 log.cc:1196] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/49cbb6122a6140f4b6a2ced60c635b77
I20260926 09:26:03.096299 10515 ts_tablet_manager.cc:1939] T 49cbb6122a6140f4b6a2ced60c635b77 P d942c4efef6149b889993a497586dac7: Deleting consensus metadata
I20260926 09:26:03.096762 10001 catalog_manager.cc:4841] TS d942c4efef6149b889993a497586dac7 (127.24.235.67:46269): tablet 49cbb6122a6140f4b6a2ced60c635b77 (table test-table [id=df9fbcdb98a74ca7a5b8e9f9520220fa]) successfully deleted
W20260926 09:26:03.097388 9999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:38039 (error 108)
I20260926 09:26:03.097630 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10211
I20260926 09:26:03.098371 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.097795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:38039 (local address 127.0.0.1:51802)
0926 09:26:03.098307 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.098343 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493}
W20260926 09:26:03.098491 9999 catalog_manager.cc:4584] TS cbc1431fe26945c1982f9281d9269eeb (127.24.235.65:38039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49cbb6122a6140f4b6a2ced60c635b77: Network error: Client connection negotiation failed: client connection to 127.24.235.65:38039: connect: Connection refused (error 111)
I20260926 09:26:03.103207 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10345
I20260926 09:26:03.108042 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 9983
2026-09-26T09:26:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T09:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:03Z Disabled control of system clock
I20260926 09:26:03.137832 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46251
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:03.221156 10521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:03.221381 10521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:03.221402 10521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:03.222613 10521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:03.222659 10521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:03.222672 10521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:03.222685 10521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:03.222697 10521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:03.224026 10521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:03.224223 10521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:03.224458 10521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:03.229513 10529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.229526 10526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.229526 10527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:03.229946 10521 server_base.cc:1034] running on GCE node
I20260926 09:26:03.230254 10521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:03.230527 10521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:03.231679 10521 hybrid_clock.cc:648] HybridClock initialized: now 1790414763231656 us; error 41 us; skew 500 ppm
I20260926 09:26:03.232969 10521 webserver.cc:460] Webserver started at http://127.24.235.126:45835/ using document root <none> and password file <none>
I20260926 09:26:03.233274 10521 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:03.233325 10521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:03.233419 10521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:03.234515 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "2cb37138572543a8925eb34fc8a43c09"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "3c2e8839eea04f2bfa6f332c51ac24b9"
server_key_iv: "6db29b7567e38aa99aa60174d6f56ee0"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.234818 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "2cb37138572543a8925eb34fc8a43c09"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "3c2e8839eea04f2bfa6f332c51ac24b9"
server_key_iv: "6db29b7567e38aa99aa60174d6f56ee0"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.236178 10521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:03.236872 10535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.237090 10521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:03.237169 10521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "2cb37138572543a8925eb34fc8a43c09"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "3c2e8839eea04f2bfa6f332c51ac24b9"
server_key_iv: "6db29b7567e38aa99aa60174d6f56ee0"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.237226 10521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:03.245998 10521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:03.246277 10521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:03.246380 10521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:03.250576 10521 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:46251
I20260926 09:26:03.250710 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:46251 every 8 connection(s)
I20260926 09:26:03.251008 10521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:03.252055 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.252310 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10521
I20260926 09:26:03.252414 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:03.252642 25517 external_mini_cluster.cc:1428] Setting key 1604a213c48a6501d04519067b860e93
I20260926 09:26:03.255790 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: Bootstrap starting.
I20260926 09:26:03.256409 10588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.256747 10588 log.cc:824] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.257558 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: No bootstrap required, opened a new log
I20260926 09:26:03.258790 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.253264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.24.235.126:46251)
0926 09:26:03.253499 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.253508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.253545 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:26:03.254038 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.254048 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.254142 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.254293 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:03.254458 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.255797 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.256665 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.257282 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.257405 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.258072 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:03.258076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:03.258077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:03.258290 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:03.258332 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:03.258347 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:03.258416 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.258477 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.258558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260926 09:26:03.259920 10588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } }
I20260926 09:26:03.260049 10588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.260071 10588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb37138572543a8925eb34fc8a43c09, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.260218 10588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } }
I20260926 09:26:03.260293 10588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:03.260334 10588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:03.260377 10588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.260952 10588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } }
I20260926 09:26:03.261091 10588 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb37138572543a8925eb34fc8a43c09; no voters:
I20260926 09:26:03.261287 10588 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:03.261409 10593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:03.261557 10588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:03.261579 10593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb37138572543a8925eb34fc8a43c09, State: Running, Role: LEADER
I20260926 09:26:03.261718 10593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } }
I20260926 09:26:03.262128 10595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb37138572543a8925eb34fc8a43c09. Latest consensus state: current_term: 1 leader_uuid: "2cb37138572543a8925eb34fc8a43c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } } }
I20260926 09:26:03.262143 10594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cb37138572543a8925eb34fc8a43c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb37138572543a8925eb34fc8a43c09" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 46251 } } }
I20260926 09:26:03.262253 10595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:03.262262 10594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:03.262526 10602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:03.263020 10602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:03.264523 10602 catalog_manager.cc:1347] Generated new cluster ID: 55a1279c307b4db3a74a270656be9c05
I20260926 09:26:03.264576 10602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:03.277516 10602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:03.278221 10602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:03.281875 10602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: Generated new TSK 0
I20260926 09:26:03.282089 10602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:03.286207 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--builtin_ntp_servers=127.24.235.84:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:03.371906 10612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:03.372125 10612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:03.372148 10612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:03.373380 10612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:03.373425 10612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:03.373451 10612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:03.374835 10612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:03.375087 10612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:03.375442 10612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:03.380914 10617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:03.380987 10612 server_base.cc:1034] running on GCE node
W20260926 09:26:03.380909 10618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.381282 10620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:03.381495 10612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:03.381753 10612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:03.382925 10612 hybrid_clock.cc:648] HybridClock initialized: now 1790414763382899 us; error 41 us; skew 500 ppm
I20260926 09:26:03.384318 10612 webserver.cc:460] Webserver started at http://127.24.235.65:46293/ using document root <none> and password file <none>
I20260926 09:26:03.384584 10612 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:03.384709 10612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:03.384862 10612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:03.386066 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "a275a93aa49f4004ad30602ff8eed0a7"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "78d327361190abfa904c2cacdada6ca2"
server_key_iv: "77d91b7c3408e5afc43da2c7a81ce89b"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.386439 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "a275a93aa49f4004ad30602ff8eed0a7"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "78d327361190abfa904c2cacdada6ca2"
server_key_iv: "77d91b7c3408e5afc43da2c7a81ce89b"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.387759 10612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:03.388551 10626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.388783 10612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:03.388890 10612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "a275a93aa49f4004ad30602ff8eed0a7"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "78d327361190abfa904c2cacdada6ca2"
server_key_iv: "77d91b7c3408e5afc43da2c7a81ce89b"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.388962 10612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:03.413924 10612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:03.414218 10612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:03.414324 10612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:03.414529 10612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:03.414928 10612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:03.414963 10612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.415020 10612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:03.415047 10612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.420928 10612 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:34795
I20260926 09:26:03.421064 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:34795 every 8 connection(s)
I20260926 09:26:03.421386 10612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:26:03.422020 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10612
I20260926 09:26:03.422140 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:03.422381 25517 external_mini_cluster.cc:1428] Setting key 52f90d1c3bba81d0ba660686f0f04688
I20260926 09:26:03.423400 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--builtin_ntp_servers=127.24.235.84:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:26:03.427651 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.422299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:45347 (local address 127.24.235.126:46251)
0926 09:26:03.422345 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.422347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.422714 (+ 367us) server_negotiation.cc:407] Connection header received
0926 09:26:03.423299 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.423301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.423317 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.423370 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:03.424138 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.424779 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.425873 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.426211 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.426286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.427277 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:03.427279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:03.427280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:03.427285 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:03.427305 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:03.427310 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:03.427374 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.427480 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.427518 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:03.427745 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.422325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46251 (local address 127.24.235.65:45347)
0926 09:26:03.422552 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.422604 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.423180 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.423452 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.423457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.423469 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:03.424038 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.424042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.424881 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.424882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.425790 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.425793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.426320 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.426321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.426444 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.427108 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:03.427109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:03.427147 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:03.427181 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:03.427381 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:03.427386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:03.427388 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.427444 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.427533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260926 09:26:03.428352 10740 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46251
I20260926 09:26:03.428464 10740 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:03.428685 10740 heartbeater.cc:511] Master 127.24.235.126:46251 requested a full tablet report, sending...
I20260926 09:26:03.429384 10552 ts_manager.cc:194] Registered new tserver with Master: a275a93aa49f4004ad30602ff8eed0a7 (127.24.235.65:34795)
I20260926 09:26:03.430388 10552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.65:45347
W20260926 09:26:03.507649 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:03.507867 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:03.507889 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:03.508997 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:03.509052 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:03.509078 10743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:03.510349 10743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:03.510533 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:03.510790 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:03.516036 10743 server_base.cc:1034] running on GCE node
W20260926 09:26:03.516042 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.516047 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.516181 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:03.516492 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:03.516781 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:03.517956 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790414763517939 us; error 35 us; skew 500 ppm
I20260926 09:26:03.519336 10743 webserver.cc:460] Webserver started at http://127.24.235.66:44659/ using document root <none> and password file <none>
I20260926 09:26:03.519590 10743 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:03.519670 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:03.519827 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:03.520939 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "9ce50553bf304680803aa85a655af0f0"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "e24831912432e349571324a2b13e95e0"
server_key_iv: "1848f1edffccbf1da3992ebf76190e57"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.521389 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "9ce50553bf304680803aa85a655af0f0"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "e24831912432e349571324a2b13e95e0"
server_key_iv: "1848f1edffccbf1da3992ebf76190e57"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.522776 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:03.523583 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.523810 10743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:03.523914 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "9ce50553bf304680803aa85a655af0f0"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "e24831912432e349571324a2b13e95e0"
server_key_iv: "1848f1edffccbf1da3992ebf76190e57"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.524001 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:03.536684 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:03.537128 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:03.537295 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:03.537555 10743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:03.537923 10743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:03.537981 10743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.538025 10743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:03.538053 10743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.544850 10743 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:35111
I20260926 09:26:03.544900 10870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:35111 every 8 connection(s)
I20260926 09:26:03.545316 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:03.549144 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10743
I20260926 09:26:03.549279 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:03.549496 25517 external_mini_cluster.cc:1428] Setting key c8621bbb0e18c9637d390e889b14bfca
I20260926 09:26:03.550688 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--builtin_ntp_servers=127.24.235.84:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:26:03.551759 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.546139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:34947 (local address 127.24.235.126:46251)
0926 09:26:03.546204 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.546206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.546463 (+ 257us) server_negotiation.cc:407] Connection header received
0926 09:26:03.547052 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.547054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.547067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.547157 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:03.547876 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.548411 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.549893 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.550263 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.550356 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.551471 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:03.551473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:03.551473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:03.551477 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:03.551494 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:03.551498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:03.551553 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.551666 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.551692 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:26:03.552152 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.546150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46251 (local address 127.24.235.66:34947)
0926 09:26:03.546334 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.546371 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.546936 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.547225 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.547229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.547238 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:03.547764 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.547768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.548526 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.548528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.549766 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.549770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.550390 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.550391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.550570 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.551265 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:03.551266 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:03.551311 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:03.551346 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:03.551575 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:03.551582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:03.551586 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.551661 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.551802 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260926 09:26:03.552763 10871 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46251
I20260926 09:26:03.552896 10871 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:03.553161 10871 heartbeater.cc:511] Master 127.24.235.126:46251 requested a full tablet report, sending...
I20260926 09:26:03.553608 10552 ts_manager.cc:194] Registered new tserver with Master: 9ce50553bf304680803aa85a655af0f0 (127.24.235.66:35111)
I20260926 09:26:03.554147 10552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.66:34947
W20260926 09:26:03.634068 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:03.634276 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:03.634297 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:03.635371 10874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:03.635403 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:03.635427 10874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:03.636682 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:03.636893 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:03.637221 10874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:03.642354 10882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.642342 10880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:03.642325 10879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:03.642539 10874 server_base.cc:1034] running on GCE node
I20260926 09:26:03.642874 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:03.643097 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:03.644248 10874 hybrid_clock.cc:648] HybridClock initialized: now 1790414763644230 us; error 29 us; skew 500 ppm
I20260926 09:26:03.645613 10874 webserver.cc:460] Webserver started at http://127.24.235.67:34421/ using document root <none> and password file <none>
I20260926 09:26:03.645865 10874 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:03.645913 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:03.646013 10874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:03.647162 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "226fd9385d494290b605aa5b96b13394"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "22784cf301e484ac2f23c204d8608bf0"
server_key_iv: "fe5bc703ff12b63f5d0740e49575de6a"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.647482 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "226fd9385d494290b605aa5b96b13394"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "22784cf301e484ac2f23c204d8608bf0"
server_key_iv: "fe5bc703ff12b63f5d0740e49575de6a"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.648859 10874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:03.649667 10888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.649840 10874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:03.649928 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "226fd9385d494290b605aa5b96b13394"
format_stamp: "Formatted at 2026-09-26 09:26:03 on dist-test-slave-7zlw"
server_key: "22784cf301e484ac2f23c204d8608bf0"
server_key_iv: "fe5bc703ff12b63f5d0740e49575de6a"
server_key_version: "encryptionkey@0"
I20260926 09:26:03.649997 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:03.658614 10874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:03.658960 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:03.659106 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:03.659333 10874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:03.659642 10874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:03.659678 10874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.659699 10874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:03.659745 10874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:03.666106 10874 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:43187
I20260926 09:26:03.666159 11001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:43187 every 8 connection(s)
I20260926 09:26:03.666505 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:03.666900 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 10874
I20260926 09:26:03.667014 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:03.667299 25517 external_mini_cluster.cc:1428] Setting key 085266d92bceae860509e82ef24aa1da
I20260926 09:26:03.672686 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.667368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:33897 (local address 127.24.235.126:46251)
0926 09:26:03.667425 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.667427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.667691 (+ 264us) server_negotiation.cc:407] Connection header received
0926 09:26:03.668533 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.668534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.668547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.668613 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:03.669464 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.670028 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.671127 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.671343 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.671437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.672412 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:03.672413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:03.672413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:03.672418 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:03.672433 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:03.672436 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:03.672490 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.672595 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.672619 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 09:26:03.672950 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.667326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:46251 (local address 127.24.235.67:33897)
0926 09:26:03.667559 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.667610 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.668411 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.668693 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.668698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.668711 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:03.669360 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.669366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.670133 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.670134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.671007 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.671009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.671451 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.671452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.671614 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.672260 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:03.672261 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:03.672297 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:03.672319 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:03.672493 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:03.672496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:03.672498 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.672594 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.672685 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 09:26:03.673527 11002 heartbeater.cc:347] Connected to a master server at 127.24.235.126:46251
I20260926 09:26:03.673611 11002 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:03.673884 11002 heartbeater.cc:511] Master 127.24.235.126:46251 requested a full tablet report, sending...
I20260926 09:26:03.674360 10552 ts_manager.cc:194] Registered new tserver with Master: 226fd9385d494290b605aa5b96b13394 (127.24.235.67:43187)
I20260926 09:26:03.674815 10552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.67:33897
I20260926 09:26:03.679167 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:03.683247 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.680517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.24.235.126:46251)
0926 09:26:03.680559 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.680561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.680734 (+ 173us) server_negotiation.cc:407] Connection header received
0926 09:26:03.680773 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.680774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.680784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.680828 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:03.680948 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.681491 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.682200 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.682381 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.682435 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.683012 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:03.683013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:03.683014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:03.683017 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:03.683031 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:03.683035 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:03.683077 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.683125 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.683144 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:26:03.684902 10552 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:03.686539 10552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:03.693063 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.688879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.24.235.65:34795)
0926 09:26:03.689043 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.689048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.689168 (+ 120us) server_negotiation.cc:407] Connection header received
0926 09:26:03.689244 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.689246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.689262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.689346 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.689560 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690711 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.691919 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.692626 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.692701 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.692773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.692904 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.692945 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260926 09:26:03.693089 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.688698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43187 (local address 127.0.0.1:40550)
0926 09:26:03.689044 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.689054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.689258 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.689632 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.689633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.689635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.689668 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.689670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690903 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.690904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.691779 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.691782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.692700 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.692701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.692786 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.692803 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.692879 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.692962 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":5,"thread_start_us":276,"threads_started":1}
I20260926 09:26:03.693334 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.689111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.24.235.67:43187)
0926 09:26:03.689466 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.689470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.689480 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:03.689517 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.689519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.689534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.689738 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.689800 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690823 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.691865 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.692585 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.692662 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.692756 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.693216 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.693256 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":84,"threads_started":1}
I20260926 09:26:03.693087 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.688698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:34795 (local address 127.0.0.1:56902)
0926 09:26:03.689110 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.689118 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.689174 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.689399 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.689403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.689406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.689468 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.689469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690795 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.690796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.691849 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.691851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.692705 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.692706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.692786 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.692804 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.692879 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.692963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":44,"spinlock_wait_cycles":768,"thread_start_us":215,"threads_started":1}
I20260926 09:26:03.694056 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.688685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35111 (local address 127.0.0.1:58154)
0926 09:26:03.688899 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.688941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.689149 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.689640 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.689641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.689644 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.689688 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.689690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690576 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.690577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.691845 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.691847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.693834 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.693835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.693899 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.693903 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.693952 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.693991 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":56,"threads_started":1}
I20260926 09:26:03.694154 10936 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 (DEFAULT_TABLE table=test-table [id=fbac01fa1525484b93ca98b67900f68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.694331 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.688776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.24.235.66:35111)
0926 09:26:03.689415 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.689420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.689430 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:03.689472 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.689474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.689490 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.689565 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.689782 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.690482 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.692817 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.693631 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.693709 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.693801 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.694204 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.694247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":107,"threads_started":1}
I20260926 09:26:03.694350 10674 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 (DEFAULT_TABLE table=test-table [id=fbac01fa1525484b93ca98b67900f68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.694545 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3f2fc396254236a5e7c1a6e2966985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.694664 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3f2fc396254236a5e7c1a6e2966985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.697568 11017 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Bootstrap starting.
I20260926 09:26:03.697665 10805 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 (DEFAULT_TABLE table=test-table [id=fbac01fa1525484b93ca98b67900f68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:03.698002 10805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3f2fc396254236a5e7c1a6e2966985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:03.698230 11017 tablet_bootstrap.cc:654] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.698530 11017 log.cc:824] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.698770 11016 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Bootstrap starting.
I20260926 09:26:03.699337 11017 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: No bootstrap required, opened a new log
I20260926 09:26:03.699457 11016 tablet_bootstrap.cc:654] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.699477 11017 ts_tablet_manager.cc:1397] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:26:03.699784 11016 log.cc:824] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.700773 11016 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: No bootstrap required, opened a new log
I20260926 09:26:03.700839 11016 ts_tablet_manager.cc:1397] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 09:26:03.701269 11020 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Bootstrap starting.
I20260926 09:26:03.701967 11020 tablet_bootstrap.cc:654] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:03.702005 11017 raft_consensus.cc:348] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.702176 11017 raft_consensus.cc:374] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.702211 11017 raft_consensus.cc:729] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a275a93aa49f4004ad30602ff8eed0a7, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.702296 11020 log.cc:824] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:03.702350 11017 consensus_queue.cc:260] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.702638 11017 ts_tablet_manager.cc:1428] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:03.702813 10740 heartbeater.cc:503] Master 127.24.235.126:46251 was elected leader, sending a full tablet report...
I20260926 09:26:03.703115 11020 tablet_bootstrap.cc:492] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: No bootstrap required, opened a new log
I20260926 09:26:03.703202 11020 ts_tablet_manager.cc:1397] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:26:03.703239 11016 raft_consensus.cc:348] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.703390 11016 raft_consensus.cc:374] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.703421 11016 raft_consensus.cc:729] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226fd9385d494290b605aa5b96b13394, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.703531 11016 consensus_queue.cc:260] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.703804 11016 ts_tablet_manager.cc:1428] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:26:03.703826 11002 heartbeater.cc:503] Master 127.24.235.126:46251 was elected leader, sending a full tablet report...
I20260926 09:26:03.705542 11020 raft_consensus.cc:348] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.705667 11020 raft_consensus.cc:374] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:03.705688 11020 raft_consensus.cc:729] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce50553bf304680803aa85a655af0f0, State: Initialized, Role: FOLLOWER
I20260926 09:26:03.705798 11020 consensus_queue.cc:260] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.706004 11020 ts_tablet_manager.cc:1428] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:03.706113 10871 heartbeater.cc:503] Master 127.24.235.126:46251 was elected leader, sending a full tablet report...
W20260926 09:26:03.795940 10872 tablet.cc:2369] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:03.900242 11022 raft_consensus.cc:482] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:03.900382 11022 raft_consensus.cc:504] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.900892 11022 leader_election.cc:288] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ce50553bf304680803aa85a655af0f0 (127.24.235.66:35111), 226fd9385d494290b605aa5b96b13394 (127.24.235.67:43187)
I20260926 09:26:03.904191 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.901173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35111 (local address 127.24.235.65:60873)
0926 09:26:03.901244 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.901258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.901314 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.901540 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.901541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.901543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.901633 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.901636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.902416 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.902417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.903309 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.903312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.903912 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.903991 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.903994 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.904045 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.904082 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 09:26:03.904199 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.901229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:60873 (local address 127.24.235.66:35111)
0926 09:26:03.901307 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.901309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.901320 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:26:03.901413 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.901414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.901425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.901500 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.901734 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.902311 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903406 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.903815 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903874 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.903881 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.904073 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.904124 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:03.904282 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.901313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:43643 (local address 127.24.235.67:43187)
0926 09:26:03.901383 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.901386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.901543 (+ 157us) server_negotiation.cc:407] Connection header received
0926 09:26:03.901618 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.901619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.901629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.901693 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:03.901959 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.902512 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903467 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.903842 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903944 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.903952 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.904137 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.904210 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:26:03.904364 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.901177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:43187 (local address 127.24.235.65:43643)
0926 09:26:03.901452 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.901462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:03.901531 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:03.901736 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:03.901737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:03.901739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:03.901843 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.901845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.902599 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.902599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.903340 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:03.903342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.903965 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.903966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:03.904048 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.904053 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:26:03.904111 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:26:03.904148 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":164,"thread_start_us":66,"threads_started":1}
I20260926 09:26:03.904604 10825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3f2fc396254236a5e7c1a6e2966985" candidate_uuid: "a275a93aa49f4004ad30602ff8eed0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce50553bf304680803aa85a655af0f0" is_pre_election: true
I20260926 09:26:03.904807 10825 raft_consensus.cc:2454] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a275a93aa49f4004ad30602ff8eed0a7 in term 0.
I20260926 09:26:03.904773 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3f2fc396254236a5e7c1a6e2966985" candidate_uuid: "a275a93aa49f4004ad30602ff8eed0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226fd9385d494290b605aa5b96b13394" is_pre_election: true
I20260926 09:26:03.904953 10956 raft_consensus.cc:2454] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a275a93aa49f4004ad30602ff8eed0a7 in term 0.
I20260926 09:26:03.905109 10629 leader_election.cc:302] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ce50553bf304680803aa85a655af0f0, a275a93aa49f4004ad30602ff8eed0a7; no voters:
I20260926 09:26:03.905272 11022 raft_consensus.cc:2781] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:03.905349 11022 raft_consensus.cc:482] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:03.905378 11022 raft_consensus.cc:3037] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.906170 11022 raft_consensus.cc:504] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.906350 11022 leader_election.cc:288] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ce50553bf304680803aa85a655af0f0 (127.24.235.66:35111), 226fd9385d494290b605aa5b96b13394 (127.24.235.67:43187)
I20260926 09:26:03.906589 10956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3f2fc396254236a5e7c1a6e2966985" candidate_uuid: "a275a93aa49f4004ad30602ff8eed0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226fd9385d494290b605aa5b96b13394"
I20260926 09:26:03.906590 10825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3f2fc396254236a5e7c1a6e2966985" candidate_uuid: "a275a93aa49f4004ad30602ff8eed0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce50553bf304680803aa85a655af0f0"
I20260926 09:26:03.906679 10956 raft_consensus.cc:3037] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.906765 10825 raft_consensus.cc:3037] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:03.907505 10956 raft_consensus.cc:2454] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a275a93aa49f4004ad30602ff8eed0a7 in term 1.
I20260926 09:26:03.907567 10825 raft_consensus.cc:2454] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a275a93aa49f4004ad30602ff8eed0a7 in term 1.
I20260926 09:26:03.907757 10630 leader_election.cc:302] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226fd9385d494290b605aa5b96b13394, a275a93aa49f4004ad30602ff8eed0a7; no voters:
I20260926 09:26:03.907882 11022 raft_consensus.cc:2781] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:03.908087 11022 raft_consensus.cc:686] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 1 LEADER]: Becoming Leader. State: Replica: a275a93aa49f4004ad30602ff8eed0a7, State: Running, Role: LEADER
I20260926 09:26:03.908226 11022 consensus_queue.cc:237] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } }
I20260926 09:26:03.909225 10552 catalog_manager.cc:5495] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a275a93aa49f4004ad30602ff8eed0a7 (127.24.235.65). New cstate: current_term: 1 leader_uuid: "a275a93aa49f4004ad30602ff8eed0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a275a93aa49f4004ad30602ff8eed0a7" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 } health_report { overall_health: UNKNOWN } } }
W20260926 09:26:03.917240 11003 tablet.cc:2369] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:03.919718 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.917346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.24.235.65:34795)
0926 09:26:03.917394 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:03.917396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:03.917407 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:26:03.917498 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:03.917498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:03.917516 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:03.917585 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:03.917704 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.918169 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.918906 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:03.919100 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:03.919158 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:03.919316 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:03.919421 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:03.919497 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:03.919516 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 09:26:03.919533 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:26:03.921659 10956 raft_consensus.cc:1261] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 1 FOLLOWER]: Refusing update from remote peer a275a93aa49f4004ad30602ff8eed0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:03.921720 10825 raft_consensus.cc:1261] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 1 FOLLOWER]: Refusing update from remote peer a275a93aa49f4004ad30602ff8eed0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:03.922073 11022 consensus_queue.cc:1035] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "226fd9385d494290b605aa5b96b13394" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 09:26:03.922231 10741 tablet.cc:2369] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:03.922259 11026 consensus_queue.cc:1035] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ce50553bf304680803aa85a655af0f0" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:03.924640 11029 mvcc.cc:205] Tried to move back new op lower bound from 7333538873020170240 to 7333538872969089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333538873020170240}]
I20260926 09:26:03.925707 11030 mvcc.cc:205] Tried to move back new op lower bound from 7333538873020170240 to 7333538872969089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333538873020170240 or (T in {7333538873020170240})}]
I20260926 09:26:03.927199 10551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:03.927302 10551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:03.928226 10551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cb37138572543a8925eb34fc8a43c09: Sending DeleteTablet for 3 replicas of tablet 1d3f2fc396254236a5e7c1a6e2966985
I20260926 09:26:03.928624 10936 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='slave'} at 127.0.0.1:40550
I20260926 09:26:03.928601 10805 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='slave'} at 127.0.0.1:58154
I20260926 09:26:03.928745 10674 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3f2fc396254236a5e7c1a6e2966985 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:03 UTC) from {username='slave'} at 127.0.0.1:56902
I20260926 09:26:03.929060 11041 tablet_replica.cc:330] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7: stopping tablet replica
I20260926 09:26:03.929123 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10612
I20260926 09:26:03.929176 11041 raft_consensus.cc:2229] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:03.929303 11041 raft_consensus.cc:2258] T 1d3f2fc396254236a5e7c1a6e2966985 P a275a93aa49f4004ad30602ff8eed0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:03.930588 11039 tablet_replica.cc:330] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: stopping tablet replica
I20260926 09:26:03.930678 11039 raft_consensus.cc:2229] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:03.930804 11039 raft_consensus.cc:2258] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:03.929170 11040 tablet_replica.cc:330] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: stopping tablet replica
I20260926 09:26:03.931209 11040 raft_consensus.cc:2229] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:03.931329 11040 raft_consensus.cc:2258] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:03.931350 11039 ts_tablet_manager.cc:1905] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:03.931779 11040 ts_tablet_manager.cc:1905] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:03.933344 11039 ts_tablet_manager.cc:1918] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:03.933434 11039 log.cc:1196] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/1d3f2fc396254236a5e7c1a6e2966985
I20260926 09:26:03.933506 11040 ts_tablet_manager.cc:1918] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:03.933580 11040 log.cc:1196] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/1d3f2fc396254236a5e7c1a6e2966985
I20260926 09:26:03.933712 11039 ts_tablet_manager.cc:1939] T 1d3f2fc396254236a5e7c1a6e2966985 P 226fd9385d494290b605aa5b96b13394: Deleting consensus metadata
I20260926 09:26:03.933840 11040 ts_tablet_manager.cc:1939] T 1d3f2fc396254236a5e7c1a6e2966985 P 9ce50553bf304680803aa85a655af0f0: Deleting consensus metadata
I20260926 09:26:03.934188 10539 catalog_manager.cc:4841] TS 226fd9385d494290b605aa5b96b13394 (127.24.235.67:43187): tablet 1d3f2fc396254236a5e7c1a6e2966985 (table test-table [id=fbac01fa1525484b93ca98b67900f68f]) successfully deleted
I20260926 09:26:03.934511 10538 catalog_manager.cc:4841] TS 9ce50553bf304680803aa85a655af0f0 (127.24.235.66:35111): tablet 1d3f2fc396254236a5e7c1a6e2966985 (table test-table [id=fbac01fa1525484b93ca98b67900f68f]) successfully deleted
W20260926 09:26:03.935159 10536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:34795 (error 108)
I20260926 09:26:03.935702 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:03.935554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:34795 (local address 127.0.0.1:56924)
0926 09:26:03.935615 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:03.935660 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:26:03.935900 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10743
W20260926 09:26:03.940913 10536 catalog_manager.cc:4584] TS a275a93aa49f4004ad30602ff8eed0a7 (127.24.235.65:34795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d3f2fc396254236a5e7c1a6e2966985: Network error: Client connection negotiation failed: client connection to 127.24.235.65:34795: connect: Connection refused (error 111)
I20260926 09:26:03.941811 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10874
I20260926 09:26:03.947970 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 10521
2026-09-26T09:26:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (838 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:04 dist-test-slave-7zlw krb5kdc[11045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:04 dist-test-slave-7zlw krb5kdc[11045](info): set up 2 sockets
Sep 26 09:26:04 dist-test-slave-7zlw krb5kdc[11045](info): commencing operation
krb5kdc: starting...
W20260926 09:26:04.993088 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:05.124763 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43573
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:05.210979 11058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:05.211197 11058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:05.211221 11058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:05.212343 11058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:05.212379 11058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:05.212393 11058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:05.212406 11058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:05.212419 11058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:05.213925 11058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:05.214148 11058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:05.214334 11058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:05.219609 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.219455 11066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.219682 11058 server_base.cc:1034] running on GCE node
W20260926 09:26:05.219455 11064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.220183 11058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:05.220500 11058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:05.221683 11058 hybrid_clock.cc:648] HybridClock initialized: now 1790414765221669 us; error 46 us; skew 500 ppm
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:05.224232 11058 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:05.224642 11058 webserver.cc:460] Webserver started at http://127.24.235.126:41691/ using document root <none> and password file <none>
I20260926 09:26:05.224886 11058 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:05.224933 11058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:05.225086 11058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:05.226181 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "939ad02a400d4d74a3c90608ca39d850"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "2f06e528ad3678672aec85867b9d47d5"
server_key_iv: "71d4d7aa07c118aa3707927501db6729"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.226557 11058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "939ad02a400d4d74a3c90608ca39d850"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "2f06e528ad3678672aec85867b9d47d5"
server_key_iv: "71d4d7aa07c118aa3707927501db6729"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.227964 11058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:05.228767 11073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.229035 11058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:05.229127 11058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "939ad02a400d4d74a3c90608ca39d850"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "2f06e528ad3678672aec85867b9d47d5"
server_key_iv: "71d4d7aa07c118aa3707927501db6729"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.229184 11058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:05.243865 11058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:05.246122 11058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:05.246263 11058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:05.250646 11058 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:43573
I20260926 09:26:05.250731 11125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:43573 every 8 connection(s)
I20260926 09:26:05.251068 11058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:05.252070 11126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:05.254448 11126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: Bootstrap starting.
I20260926 09:26:05.255065 11126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:05.255368 11126 log.cc:824] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:05.256083 11126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: No bootstrap required, opened a new log
I20260926 09:26:05.257802 11126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } }
I20260926 09:26:05.257943 11126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:05.257966 11126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939ad02a400d4d74a3c90608ca39d850, State: Initialized, Role: FOLLOWER
I20260926 09:26:05.258149 11126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } }
I20260926 09:26:05.258215 11126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:05.258236 11126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:05.258306 11126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:05.258841 11126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } }
I20260926 09:26:05.258955 11126 leader_election.cc:302] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 939ad02a400d4d74a3c90608ca39d850; no voters:
I20260926 09:26:05.259150 11126 leader_election.cc:288] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:05.259189 11129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:05.259395 11129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [term 1 LEADER]: Becoming Leader. State: Replica: 939ad02a400d4d74a3c90608ca39d850, State: Running, Role: LEADER
I20260926 09:26:05.259423 11126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:05.259488 11129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } }
I20260926 09:26:05.260049 11131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 939ad02a400d4d74a3c90608ca39d850. Latest consensus state: current_term: 1 leader_uuid: "939ad02a400d4d74a3c90608ca39d850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } } }
I20260926 09:26:05.260138 11131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:05.260206 11130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "939ad02a400d4d74a3c90608ca39d850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "939ad02a400d4d74a3c90608ca39d850" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43573 } } }
I20260926 09:26:05.260298 11130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:05.260421 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11058
I20260926 09:26:05.260537 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:05.260789 25517 external_mini_cluster.cc:1428] Setting key 052ccf02871c524d00c6afac51b76dff
I20260926 09:26:05.260792 11138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:05.261395 11138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:05.263041 11138 catalog_manager.cc:1347] Generated new cluster ID: 97a3c4852df14885a3d712f8c8cb69d5
I20260926 09:26:05.263094 11138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:05.271529 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.261533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.24.235.126:43573)
0926 09:26:05.261837 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.261846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.261945 (+ 99us) server_negotiation.cc:407] Connection header received
0926 09:26:05.262076 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.262084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.262144 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.262287 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:05.263086 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.264065 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.264870 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.265381 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.265500 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.267936 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:05.267941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:05.267941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:05.267954 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:05.269847 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.270272 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.270275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.270276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.270396 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.270620 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.270621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.270621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.270881 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:05.271055 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.271164 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.271317 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":66,"threads_started":1}
I20260926 09:26:05.272958 11138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:05.273562 11138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:05.282966 11138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: Generated new TSK 0
I20260926 09:26:05.283212 11138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:26:05.319976 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:05.424103 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:05.424325 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:05.424351 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:05.425583 11152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:05.425659 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:05.425706 11152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:05.427214 11152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:05.427426 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:05.427629 11152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:05.432927 11157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.432928 11152 server_base.cc:1034] running on GCE node
W20260926 09:26:05.432906 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.432909 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.433395 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:05.433697 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:05.434893 11152 hybrid_clock.cc:648] HybridClock initialized: now 1790414765434870 us; error 40 us; skew 500 ppm
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:05.437693 11152 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:26:05.438100 11152 webserver.cc:460] Webserver started at http://127.24.235.65:36933/ using document root <none> and password file <none>
I20260926 09:26:05.438347 11152 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:05.438395 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:05.438539 11152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:05.439635 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "37bff887678348f68670291303130d58"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "1096a799d6c1375691e1e4dfa860ce8e"
server_key_iv: "185ee7ce23b1c515f4c8e612c4ad8f7b"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.440011 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "37bff887678348f68670291303130d58"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "1096a799d6c1375691e1e4dfa860ce8e"
server_key_iv: "185ee7ce23b1c515f4c8e612c4ad8f7b"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.441466 11152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:05.442595 11167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.442849 11152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:05.442979 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "37bff887678348f68670291303130d58"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "1096a799d6c1375691e1e4dfa860ce8e"
server_key_iv: "185ee7ce23b1c515f4c8e612c4ad8f7b"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.443068 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:05.458992 11152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:05.461818 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:05.461985 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:05.462247 11152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:05.462566 11152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:05.462621 11152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.462657 11152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:05.462689 11152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.469046 11152 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:42203
I20260926 09:26:05.469151 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:42203 every 8 connection(s)
I20260926 09:26:05.469491 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:05.476567 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11152
I20260926 09:26:05.476706 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:05.476933 25517 external_mini_cluster.cc:1428] Setting key 3abc8db3fceb1d7cbbcbcef5824ae4a4
I20260926 09:26:05.479470 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.470277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:37761 (local address 127.24.235.126:43573)
0926 09:26:05.470320 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.470322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.470513 (+ 191us) server_negotiation.cc:407] Connection header received
0926 09:26:05.471019 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.471020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.471029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.471085 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:05.472084 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.472604 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.473714 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.473913 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.473971 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.476179 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:05.476180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:05.476181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:05.476190 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:05.477992 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.478376 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.478377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.478378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.478444 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.478736 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.478737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.478737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.478894 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:05.478969 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.479227 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.479335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:26:05.479504 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.470200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43573 (local address 127.24.235.65:37761)
0926 09:26:05.470388 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.470432 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.470932 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.471151 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.471154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.471753 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:05.471978 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.471983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.472702 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.472702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.473616 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.473619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.473994 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.473995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.474098 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.474620 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:05.474628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:05.476078 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:05.478068 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.478075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.478077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.478299 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.478518 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.478518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.478519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.478623 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.478964 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:05.478970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:05.479089 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.479200 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.479332 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260926 09:26:05.480106 11281 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43573
I20260926 09:26:05.480257 11281 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:05.480465 11281 heartbeater.cc:511] Master 127.24.235.126:43573 requested a full tablet report, sending...
I20260926 09:26:05.481071 11090 ts_manager.cc:194] Registered new tserver with Master: 37bff887678348f68670291303130d58 (127.24.235.65:42203)
I20260926 09:26:05.481691 11090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:37761
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:26:05.504951 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:05.592942 11286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:05.593174 11286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:05.593200 11286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:05.594547 11286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:05.594597 11286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:05.594627 11286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:05.596307 11286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:05.596506 11286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:05.596686 11286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:05.601976 11294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.601989 11292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.602049 11291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.602161 11286 server_base.cc:1034] running on GCE node
I20260926 09:26:05.602392 11286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:05.602650 11286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:05.603812 11286 hybrid_clock.cc:648] HybridClock initialized: now 1790414765603807 us; error 37 us; skew 500 ppm
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:05.606357 11286 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:26:05.606827 11286 webserver.cc:460] Webserver started at http://127.24.235.66:33097/ using document root <none> and password file <none>
I20260926 09:26:05.607067 11286 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:05.607136 11286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:05.607291 11286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:05.608341 11286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "3628ac25f2a746aab3acbedb3b30ca08"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "6296e26cfdccfa0f5f730cb2a7298dd5"
server_key_iv: "3289dc09e64b607d0c15ccc82cc3c2f5"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.608700 11286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "3628ac25f2a746aab3acbedb3b30ca08"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "6296e26cfdccfa0f5f730cb2a7298dd5"
server_key_iv: "3289dc09e64b607d0c15ccc82cc3c2f5"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.610322 11286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:05.611068 11301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.611239 11286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.611361 11286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "3628ac25f2a746aab3acbedb3b30ca08"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "6296e26cfdccfa0f5f730cb2a7298dd5"
server_key_iv: "3289dc09e64b607d0c15ccc82cc3c2f5"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.611444 11286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:05.628239 11286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:05.630399 11286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:05.630563 11286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:05.630812 11286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:05.631124 11286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:05.631179 11286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.631218 11286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:05.631239 11286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.637913 11286 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:40669
I20260926 09:26:05.638003 11414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:40669 every 8 connection(s)
I20260926 09:26:05.638335 11286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:05.640700 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11286
I20260926 09:26:05.640823 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:05.641070 25517 external_mini_cluster.cc:1428] Setting key 48bcc846d7e6d025755926988d03a7ff
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:05.648581 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.639250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:45161 (local address 127.24.235.126:43573)
0926 09:26:05.639339 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.639341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.639592 (+ 251us) server_negotiation.cc:407] Connection header received
0926 09:26:05.640018 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.640019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.640031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.640106 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:05.641119 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.641671 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.642703 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.642911 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.643003 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.645514 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:05.645515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:05.645516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:05.645523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:05.647007 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.647601 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.647601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.647602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.647652 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.647943 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.647944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.647944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.648047 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:05.648094 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.648447 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.648510 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 09:26:05.648878 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.639225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43573 (local address 127.24.235.66:45161)
0926 09:26:05.639394 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.639436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.639926 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.640163 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.640167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.640745 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:05.641038 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.641043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.641779 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.641781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.642608 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.642611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.643015 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.643016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.643155 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.643744 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:05.643752 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:05.645388 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:05.647162 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.647170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.647172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.647523 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.647719 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.647719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.647720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.647867 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.648113 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:05.648120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:05.648301 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.648429 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.648619 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260926 09:26:05.649502 11415 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43573
I20260926 09:26:05.649587 11415 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:05.649783 11415 heartbeater.cc:511] Master 127.24.235.126:43573 requested a full tablet report, sending...
I20260926 09:26:05.650203 11090 ts_manager.cc:194] Registered new tserver with Master: 3628ac25f2a746aab3acbedb3b30ca08 (127.24.235.66:40669)
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
I20260926 09:26:05.650621 11090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:45161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:26:05.669864 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:05.753273 11420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:05.753481 11420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:05.753506 11420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:05.754590 11420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:05.754664 11420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:05.754711 11420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:05.756145 11420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:05.756349 11420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:05.756523 11420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:05.761824 11425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.761822 11426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:05.761798 11428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:05.762173 11420 server_base.cc:1034] running on GCE node
I20260926 09:26:05.762429 11420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:05.762727 11420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:05.763895 11420 hybrid_clock.cc:648] HybridClock initialized: now 1790414765763882 us; error 26 us; skew 500 ppm
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:05.766605 11420 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:26:05.767128 11420 webserver.cc:460] Webserver started at http://127.24.235.67:35843/ using document root <none> and password file <none>
I20260926 09:26:05.767314 11420 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:05.767354 11420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:05.767444 11420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:05.768488 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "30567bb55bca44f9877669b4b75023fc"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "8ace1a42e954c3d9209d86a7030701ec"
server_key_iv: "fbaba4090816b6cde4bd195c95e4d4ce"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.768792 11420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "30567bb55bca44f9877669b4b75023fc"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "8ace1a42e954c3d9209d86a7030701ec"
server_key_iv: "fbaba4090816b6cde4bd195c95e4d4ce"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.770143 11420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:05.770812 11435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.770990 11420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:05.771097 11420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "30567bb55bca44f9877669b4b75023fc"
format_stamp: "Formatted at 2026-09-26 09:26:05 on dist-test-slave-7zlw"
server_key: "8ace1a42e954c3d9209d86a7030701ec"
server_key_iv: "fbaba4090816b6cde4bd195c95e4d4ce"
server_key_version: "encryptionkey@0"
I20260926 09:26:05.771176 11420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:05.789865 11420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:05.792224 11420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:05.792392 11420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:05.792641 11420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:05.792953 11420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:05.793007 11420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.793084 11420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:05.793116 11420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:05.799474 11420 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:45355
I20260926 09:26:05.799511 11548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:45355 every 8 connection(s)
I20260926 09:26:05.799855 11420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:05.805457 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11420
I20260926 09:26:05.805536 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:05.805722 25517 external_mini_cluster.cc:1428] Setting key a0e43068c37ee9f30ab7ac8d292d2bc6
Sep 26 09:26:05 dist-test-slave-7zlw krb5kdc[11045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414765, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:05.809446 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.800637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:57801 (local address 127.24.235.126:43573)
0926 09:26:05.800681 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.800683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.800983 (+ 300us) server_negotiation.cc:407] Connection header received
0926 09:26:05.801558 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.801559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.801572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.801628 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:05.802626 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.803124 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.804166 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.804366 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.804458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.806935 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:05.806936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:05.806936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:05.806942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:05.808182 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.808590 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.808591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.808591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.808643 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.808869 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.808869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.808870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.808972 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:05.809090 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.809322 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.809383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:05.809602 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.800609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43573 (local address 127.24.235.67:57801)
0926 09:26:05.800835 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.800886 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.801446 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.801700 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.801704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.802289 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:05.802483 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.802488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.803226 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.803227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.804054 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.804056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.804483 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.804484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.804605 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.805119 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:05.805127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:05.806838 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:05.808279 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.808286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.808289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.808505 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.808710 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:05.808710 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:05.808710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:05.808809 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:05.809071 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:05.809078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:05.809206 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.809290 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.809404 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260926 09:26:05.810138 11549 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43573
I20260926 09:26:05.810220 11549 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:05.810353 11549 heartbeater.cc:511] Master 127.24.235.126:43573 requested a full tablet report, sending...
I20260926 09:26:05.810665 11090 ts_manager.cc:194] Registered new tserver with Master: 30567bb55bca44f9877669b4b75023fc (127.24.235.67:45355)
I20260926 09:26:05.811049 11090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:57801
I20260926 09:26:05.817585 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:05.825253 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.818992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.24.235.126:43573)
0926 09:26:05.819093 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.819095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.819133 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:26:05.819191 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.819192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.819201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.819251 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:05.819816 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.820289 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.820984 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.821241 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.821297 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.822764 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:05.822765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:05.822765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:05.822770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:05.824023 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.824445 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.824446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.824446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.824497 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:05.824694 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:05.824694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:05.824694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:05.824834 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:05.824913 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.825096 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.825148 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 09:26:05.827149 11090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:05.828876 11090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:05.835887 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.831340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40669 (local address 127.0.0.1:52386)
0926 09:26:05.831652 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.831663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.831721 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.832139 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.832140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.832142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:05.832198 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.832199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833613 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.833614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.834906 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.834908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835669 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.835732 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835736 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.835780 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.835810 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":52,"thread_start_us":118,"threads_started":1}
I20260926 09:26:05.835997 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.831569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:45355 (local address 127.0.0.1:33736)
0926 09:26:05.831697 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.831707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.831750 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.832486 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.832487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.832489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:05.832523 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.832524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833648 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.833649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.834892 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.834895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835749 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.835812 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835817 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.835875 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.835921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":51,"thread_start_us":91,"threads_started":1}
I20260926 09:26:05.836253 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.831751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.24.235.66:40669)
0926 09:26:05.831971 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.831975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.831985 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:05.832022 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.832024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.832041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.832134 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:05.832341 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833498 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.834999 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835480 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835568 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835637 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.836124 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.836170 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260926 09:26:05.836253 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.831441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.24.235.65:42203)
0926 09:26:05.831595 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.831607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.831620 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:26:05.831818 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.831822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.831842 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.831919 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:05.832152 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833374 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.834765 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835467 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835556 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835623 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.835934 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.836167 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260926 09:26:05.836263 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.832040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.24.235.67:45355)
0926 09:26:05.832317 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.832322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.832332 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:05.832371 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.832373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.832387 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.832591 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:05.832650 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833564 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.834986 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835650 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835748 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835801 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.835889 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.835928 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
I20260926 09:26:05.835886 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.831309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42203 (local address 127.0.0.1:36582)
0926 09:26:05.831516 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.831559 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.831634 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.831948 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.831951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.831954 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:05.832063 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.832066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.833471 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.833472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.834666 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.834668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.835566 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.835567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.835647 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.835663 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.835731 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.835794 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":92,"thread_start_us":92,"threads_started":1}
I20260926 09:26:05.837147 11483 tablet_service.cc:1467] Processing CreateTablet for tablet f62fed5123324f1d8b666046c14d7856 (DEFAULT_TABLE table=test-table [id=b7d8fd4708fb4f0a8465c29df5e3f0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:05.837212 11349 tablet_service.cc:1467] Processing CreateTablet for tablet f62fed5123324f1d8b666046c14d7856 (DEFAULT_TABLE table=test-table [id=b7d8fd4708fb4f0a8465c29df5e3f0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:05.837517 11483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62fed5123324f1d8b666046c14d7856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:05.837527 11349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62fed5123324f1d8b666046c14d7856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:05.840128 11567 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Bootstrap starting.
I20260926 09:26:05.840406 11566 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Bootstrap starting.
I20260926 09:26:05.840582 11215 tablet_service.cc:1467] Processing CreateTablet for tablet f62fed5123324f1d8b666046c14d7856 (DEFAULT_TABLE table=test-table [id=b7d8fd4708fb4f0a8465c29df5e3f0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:05.840904 11567 tablet_bootstrap.cc:654] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:05.840910 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62fed5123324f1d8b666046c14d7856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:05.841094 11566 tablet_bootstrap.cc:654] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:05.841548 11567 log.cc:824] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:05.841604 11566 log.cc:824] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:05.842363 11567 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: No bootstrap required, opened a new log
I20260926 09:26:05.842372 11566 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: No bootstrap required, opened a new log
I20260926 09:26:05.842439 11566 ts_tablet_manager.cc:1397] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:26:05.842439 11567 ts_tablet_manager.cc:1397] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:26:05.844031 11570 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Bootstrap starting.
I20260926 09:26:05.844477 11566 raft_consensus.cc:348] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.844477 11567 raft_consensus.cc:348] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.844633 11570 tablet_bootstrap.cc:654] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:05.844666 11567 raft_consensus.cc:374] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:05.844666 11566 raft_consensus.cc:374] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:05.844712 11566 raft_consensus.cc:729] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30567bb55bca44f9877669b4b75023fc, State: Initialized, Role: FOLLOWER
I20260926 09:26:05.844709 11567 raft_consensus.cc:729] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3628ac25f2a746aab3acbedb3b30ca08, State: Initialized, Role: FOLLOWER
I20260926 09:26:05.844832 11566 consensus_queue.cc:260] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.844832 11567 consensus_queue.cc:260] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.845130 11566 ts_tablet_manager.cc:1428] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:26:05.845158 11567 ts_tablet_manager.cc:1428] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:05.845260 11549 heartbeater.cc:503] Master 127.24.235.126:43573 was elected leader, sending a full tablet report...
I20260926 09:26:05.845265 11415 heartbeater.cc:503] Master 127.24.235.126:43573 was elected leader, sending a full tablet report...
I20260926 09:26:05.845332 11570 log.cc:824] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:05.846215 11570 tablet_bootstrap.cc:492] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: No bootstrap required, opened a new log
I20260926 09:26:05.846292 11570 ts_tablet_manager.cc:1397] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 09:26:05.847892 11570 raft_consensus.cc:348] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.848057 11570 raft_consensus.cc:374] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:05.848080 11570 raft_consensus.cc:729] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bff887678348f68670291303130d58, State: Initialized, Role: FOLLOWER
I20260926 09:26:05.848183 11570 consensus_queue.cc:260] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.848425 11570 ts_tablet_manager.cc:1428] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 09:26:05.848483 11281 heartbeater.cc:503] Master 127.24.235.126:43573 was elected leader, sending a full tablet report...
W20260926 09:26:05.889145 11416 tablet.cc:2369] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:05.960354 11571 raft_consensus.cc:482] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:05.960522 11571 raft_consensus.cc:504] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.960937 11571 leader_election.cc:288] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37bff887678348f68670291303130d58 (127.24.235.65:42203), 3628ac25f2a746aab3acbedb3b30ca08 (127.24.235.66:40669)
I20260926 09:26:05.964978 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.961239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:39651 (local address 127.24.235.65:42203)
0926 09:26:05.961290 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.961292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.961419 (+ 127us) server_negotiation.cc:407] Connection header received
0926 09:26:05.961555 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.961556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.961579 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.961642 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:05.961806 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.962491 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964023 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.964608 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.964697 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.964866 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.964907 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 09:26:05.964978 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.961235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:58667 (local address 127.24.235.66:40669)
0926 09:26:05.961328 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.961330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.961338 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:26:05.961388 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.961389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.961401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.961453 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:05.961690 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.962485 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964023 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.964608 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964689 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.964697 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.964866 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.964904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 09:26:05.965005 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.961127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:40669 (local address 127.24.235.67:58667)
0926 09:26:05.961171 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.961183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.961235 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.961473 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.961474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.961476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:05.961562 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.961563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.962579 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.962579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.963919 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.963922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.964698 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.964783 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.964785 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.964844 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.964890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":60}
I20260926 09:26:05.965034 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.961151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42203 (local address 127.24.235.67:39651)
0926 09:26:05.961348 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:05.961357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:05.961434 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:05.961657 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:05.961658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:05.961660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:05.961723 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.961730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.962577 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.962578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.963919 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:05.963922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.964698 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.964699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:05.964780 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.964784 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:05.964844 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 09:26:05.964890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260926 09:26:05.965483 11369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62fed5123324f1d8b666046c14d7856" candidate_uuid: "30567bb55bca44f9877669b4b75023fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3628ac25f2a746aab3acbedb3b30ca08" is_pre_election: true
I20260926 09:26:05.965492 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62fed5123324f1d8b666046c14d7856" candidate_uuid: "30567bb55bca44f9877669b4b75023fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bff887678348f68670291303130d58" is_pre_election: true
I20260926 09:26:05.965653 11235 raft_consensus.cc:2454] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30567bb55bca44f9877669b4b75023fc in term 0.
I20260926 09:26:05.965653 11369 raft_consensus.cc:2454] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30567bb55bca44f9877669b4b75023fc in term 0.
I20260926 09:26:05.965878 11436 leader_election.cc:302] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30567bb55bca44f9877669b4b75023fc, 3628ac25f2a746aab3acbedb3b30ca08; no voters:
I20260926 09:26:05.966001 11571 raft_consensus.cc:2781] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:05.966056 11571 raft_consensus.cc:482] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:05.966078 11571 raft_consensus.cc:3037] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:05.966820 11571 raft_consensus.cc:504] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.966971 11571 leader_election.cc:288] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [CANDIDATE]: Term 1 election: Requested vote from peers 37bff887678348f68670291303130d58 (127.24.235.65:42203), 3628ac25f2a746aab3acbedb3b30ca08 (127.24.235.66:40669)
I20260926 09:26:05.967180 11369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62fed5123324f1d8b666046c14d7856" candidate_uuid: "30567bb55bca44f9877669b4b75023fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3628ac25f2a746aab3acbedb3b30ca08"
I20260926 09:26:05.967166 11235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62fed5123324f1d8b666046c14d7856" candidate_uuid: "30567bb55bca44f9877669b4b75023fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37bff887678348f68670291303130d58"
I20260926 09:26:05.967255 11369 raft_consensus.cc:3037] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:05.967255 11235 raft_consensus.cc:3037] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:05.968194 11369 raft_consensus.cc:2454] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30567bb55bca44f9877669b4b75023fc in term 1.
I20260926 09:26:05.968289 11235 raft_consensus.cc:2454] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30567bb55bca44f9877669b4b75023fc in term 1.
I20260926 09:26:05.968447 11439 leader_election.cc:302] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30567bb55bca44f9877669b4b75023fc, 37bff887678348f68670291303130d58; no voters:
I20260926 09:26:05.968540 11571 raft_consensus.cc:2781] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:05.968691 11571 raft_consensus.cc:686] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 1 LEADER]: Becoming Leader. State: Replica: 30567bb55bca44f9877669b4b75023fc, State: Running, Role: LEADER
I20260926 09:26:05.968767 11571 consensus_queue.cc:237] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } }
I20260926 09:26:05.969719 11089 catalog_manager.cc:5495] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 30567bb55bca44f9877669b4b75023fc (127.24.235.67). New cstate: current_term: 1 leader_uuid: "30567bb55bca44f9877669b4b75023fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30567bb55bca44f9877669b4b75023fc" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 } health_report { overall_health: UNKNOWN } } }
W20260926 09:26:05.970237 11282 tablet.cc:2369] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:05.981799 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:05.979110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.24.235.67:45355)
0926 09:26:05.979178 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:05.979180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:05.979189 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:05.979225 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:05.979226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:05.979250 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:05.979335 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:05.979488 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.980012 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.980820 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:05.981118 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:05.981211 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:05.981388 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:05.981496 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:05.981616 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:05.981674 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 09:26:05.981685 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:05.983608 11235 raft_consensus.cc:1261] T f62fed5123324f1d8b666046c14d7856 P 37bff887678348f68670291303130d58 [term 1 FOLLOWER]: Refusing update from remote peer 30567bb55bca44f9877669b4b75023fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:05.983605 11369 raft_consensus.cc:1261] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 1 FOLLOWER]: Refusing update from remote peer 30567bb55bca44f9877669b4b75023fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:05.983981 11571 consensus_queue.cc:1035] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3628ac25f2a746aab3acbedb3b30ca08" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:05.984118 11576 consensus_queue.cc:1035] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37bff887678348f68670291303130d58" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:05.986631 11580 mvcc.cc:205] Tried to move back new op lower bound from 7333538881466019840 to 7333538881408946176. Current Snapshot: MvccSnapshot[applied={T|T < 7333538881466019840}]
I20260926 09:26:05.987458 11581 mvcc.cc:205] Tried to move back new op lower bound from 7333538881466019840 to 7333538881408946176. Current Snapshot: MvccSnapshot[applied={T|T < 7333538881466019840}]
I20260926 09:26:05.989142 11089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:05.989248 11089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:05.990002 11089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 939ad02a400d4d74a3c90608ca39d850: Sending DeleteTablet for 3 replicas of tablet f62fed5123324f1d8b666046c14d7856
I20260926 09:26:05.990351 11215 tablet_service.cc:1514] Processing DeleteTablet for tablet f62fed5123324f1d8b666046c14d7856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:05 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:36582
I20260926 09:26:05.990348 11483 tablet_service.cc:1514] Processing DeleteTablet for tablet f62fed5123324f1d8b666046c14d7856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:05 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:33736
I20260926 09:26:05.990573 11588 tablet_replica.cc:330] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: stopping tablet replica
I20260926 09:26:05.990572 11349 tablet_service.cc:1514] Processing DeleteTablet for tablet f62fed5123324f1d8b666046c14d7856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:05 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:52386
I20260926 09:26:05.990690 11588 raft_consensus.cc:2229] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:05.990825 11588 raft_consensus.cc:2258] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:05.990829 11590 tablet_replica.cc:330] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: stopping tablet replica
I20260926 09:26:05.991021 11590 raft_consensus.cc:2229] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:05.991096 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11152
I20260926 09:26:05.991237 11590 raft_consensus.cc:2258] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:05.991285 11588 ts_tablet_manager.cc:1905] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:05.992148 11590 ts_tablet_manager.cc:1905] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:05.993366 11588 ts_tablet_manager.cc:1918] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:05.993446 11588 log.cc:1196] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/f62fed5123324f1d8b666046c14d7856
I20260926 09:26:05.993737 11588 ts_tablet_manager.cc:1939] T f62fed5123324f1d8b666046c14d7856 P 30567bb55bca44f9877669b4b75023fc: Deleting consensus metadata
I20260926 09:26:05.994154 11590 ts_tablet_manager.cc:1918] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:05.994237 11590 log.cc:1196] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/f62fed5123324f1d8b666046c14d7856
I20260926 09:26:05.994544 11590 ts_tablet_manager.cc:1939] T f62fed5123324f1d8b666046c14d7856 P 3628ac25f2a746aab3acbedb3b30ca08: Deleting consensus metadata
I20260926 09:26:05.995111 11074 catalog_manager.cc:4841] TS 3628ac25f2a746aab3acbedb3b30ca08 (127.24.235.66:40669): tablet f62fed5123324f1d8b666046c14d7856 (table test-table [id=b7d8fd4708fb4f0a8465c29df5e3f0e9]) successfully deleted
I20260926 09:26:05.995327 11077 catalog_manager.cc:4841] TS 30567bb55bca44f9877669b4b75023fc (127.24.235.67:45355): tablet f62fed5123324f1d8b666046c14d7856 (table test-table [id=b7d8fd4708fb4f0a8465c29df5e3f0e9]) successfully deleted
W20260926 09:26:05.999290 11077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:42203 (error 108)
I20260926 09:26:05.999497 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11286
I20260926 09:26:06.000538 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.000366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42203 (local address 127.0.0.1:36590)
0926 09:26:06.000444 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.000497 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:42203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 09:26:06.000736 11077 catalog_manager.cc:4584] TS 37bff887678348f68670291303130d58 (127.24.235.65:42203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f62fed5123324f1d8b666046c14d7856: Network error: Client connection negotiation failed: client connection to 127.24.235.65:42203: connect: Connection refused (error 111)
I20260926 09:26:06.005541 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11420
I20260926 09:26:06.010263 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11058
2026-09-26T09:26:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2062 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T09:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:06Z Disabled control of system clock
I20260926 09:26:06.039465 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43217
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:41807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:06.122488 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:06.122714 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:06.122735 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:06.124195 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:06.124254 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:06.124267 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:06.124281 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:06.124302 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:06.125886 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:06.126156 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:06.126494 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:06.131425 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.131414 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.131601 11595 server_base.cc:1034] running on GCE node
W20260926 09:26:06.131414 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.131968 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:06.132297 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:06.133448 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790414766133437 us; error 54 us; skew 500 ppm
I20260926 09:26:06.139513 11595 webserver.cc:460] Webserver started at http://127.24.235.126:43155/ using document root <none> and password file <none>
I20260926 09:26:06.139829 11595 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:06.139909 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:06.140064 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:06.141310 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "87f32d1777ec43cfbc11c945c78799fe"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "36a179a3acd2777cf11f52ec8c564e4f"
server_key_iv: "93d231f9244ed8ddbf6651bfeb42813a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.141686 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "87f32d1777ec43cfbc11c945c78799fe"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "36a179a3acd2777cf11f52ec8c564e4f"
server_key_iv: "93d231f9244ed8ddbf6651bfeb42813a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.143014 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:06.143790 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.144028 11595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:06.144124 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "87f32d1777ec43cfbc11c945c78799fe"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "36a179a3acd2777cf11f52ec8c564e4f"
server_key_iv: "93d231f9244ed8ddbf6651bfeb42813a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.144228 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:06.163798 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:06.164165 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:06.164326 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:06.168677 11595 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:43217
I20260926 09:26:06.168735 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:43217 every 8 connection(s)
I20260926 09:26:06.169131 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:06.170163 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:06.172997 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: Bootstrap starting.
I20260926 09:26:06.173840 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:06.174193 11662 log.cc:824] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:06.174606 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11595
I20260926 09:26:06.174736 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:06.174997 25517 external_mini_cluster.cc:1428] Setting key 1c8b538986f85d56db3578c6a67c6465
I20260926 09:26:06.175096 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: No bootstrap required, opened a new log
I20260926 09:26:06.176976 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } }
I20260926 09:26:06.177229 11662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:06.177279 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f32d1777ec43cfbc11c945c78799fe, State: Initialized, Role: FOLLOWER
I20260926 09:26:06.177407 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } }
I20260926 09:26:06.177502 11662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:06.177547 11662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:06.177597 11662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:06.178150 11662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } }
I20260926 09:26:06.178274 11662 leader_election.cc:302] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f32d1777ec43cfbc11c945c78799fe; no voters:
I20260926 09:26:06.178489 11662 leader_election.cc:288] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:06.178520 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:06.178630 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [term 1 LEADER]: Becoming Leader. State: Replica: 87f32d1777ec43cfbc11c945c78799fe, State: Running, Role: LEADER
I20260926 09:26:06.178716 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } }
I20260926 09:26:06.178778 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:06.179060 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f32d1777ec43cfbc11c945c78799fe. Latest consensus state: current_term: 1 leader_uuid: "87f32d1777ec43cfbc11c945c78799fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } } }
I20260926 09:26:06.179061 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f32d1777ec43cfbc11c945c78799fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f32d1777ec43cfbc11c945c78799fe" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43217 } } }
I20260926 09:26:06.179275 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:06.179491 11674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:06.180037 11674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:06.179515 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:06.182087 11674 catalog_manager.cc:1347] Generated new cluster ID: 5243307933f8404a94323648b7e85c76
I20260926 09:26:06.182144 11674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:06.186090 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.175886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.24.235.126:43217)
0926 09:26:06.176103 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.176111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.176151 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:26:06.176960 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.176968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.177088 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.177264 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:06.177481 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.179325 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.180712 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.181390 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.181532 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.185330 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:06.185335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:06.185335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:06.185536 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:06.185573 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:06.185585 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:06.185678 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.185762 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.185856 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260926 09:26:06.203848 11674 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:06.204461 11674 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:06.209116 11674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: Generated new TSK 0
I20260926 09:26:06.209322 11674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:06.218295 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--builtin_ntp_servers=127.24.235.84:41807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:06.302055 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:06.302305 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:06.302340 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:06.303544 11686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:06.303613 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:06.303653 11686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:06.305122 11686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:06.305394 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:06.305717 11686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:06.311115 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.311218 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.311435 11686 server_base.cc:1034] running on GCE node
W20260926 09:26:06.311126 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.311728 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:06.312043 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:06.313213 11686 hybrid_clock.cc:648] HybridClock initialized: now 1790414766313195 us; error 34 us; skew 500 ppm
I20260926 09:26:06.314934 11686 webserver.cc:460] Webserver started at http://127.24.235.65:34121/ using document root <none> and password file <none>
I20260926 09:26:06.315240 11686 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:06.315321 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:06.315475 11686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:06.316579 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "db557ea6e7024021aa048b3165742462"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "3643227c99e5ca3486441d8a9192d083"
server_key_iv: "afa5d8262d3e1bc3bb4e4983345ef0a4"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.316892 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "db557ea6e7024021aa048b3165742462"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "3643227c99e5ca3486441d8a9192d083"
server_key_iv: "afa5d8262d3e1bc3bb4e4983345ef0a4"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.318301 11686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:06.319027 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.319327 11686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:06.319468 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "db557ea6e7024021aa048b3165742462"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "3643227c99e5ca3486441d8a9192d083"
server_key_iv: "afa5d8262d3e1bc3bb4e4983345ef0a4"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.319572 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:06.331386 11686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:06.331758 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:06.331929 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:06.332212 11686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:06.332581 11686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:06.332652 11686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.332700 11686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:06.332733 11686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.339633 11686 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:42207
I20260926 09:26:06.339675 11813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:42207 every 8 connection(s)
I20260926 09:26:06.340108 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:26:06.344110 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11686
I20260926 09:26:06.344254 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:06.344480 25517 external_mini_cluster.cc:1428] Setting key 1c690856b3cfe01eac6e37a0bbb8faa9
I20260926 09:26:06.345598 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--builtin_ntp_servers=127.24.235.84:41807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:26:06.346046 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.340914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:35671 (local address 127.24.235.126:43217)
0926 09:26:06.340989 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.340991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.341271 (+ 280us) server_negotiation.cc:407] Connection header received
0926 09:26:06.341797 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.341799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.341812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.341879 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:06.342521 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.343020 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.344269 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.344502 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.344601 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.345707 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:06.345709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:06.345709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:06.345714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:06.345735 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:06.345741 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:06.345808 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.345934 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.345964 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:06.346306 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.340912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43217 (local address 127.24.235.65:35671)
0926 09:26:06.341132 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.341175 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.341686 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.341931 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.341934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.341942 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:06.342447 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.342451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.343112 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.343113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.344120 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.344123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.344669 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.344671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.344842 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.345509 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:06.345510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:06.345554 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:06.345585 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:06.345836 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:06.345846 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:06.345854 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.345938 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.346049 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260926 09:26:06.346751 11814 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43217
I20260926 09:26:06.346861 11814 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:06.347092 11814 heartbeater.cc:511] Master 127.24.235.126:43217 requested a full tablet report, sending...
I20260926 09:26:06.347829 11626 ts_manager.cc:194] Registered new tserver with Master: db557ea6e7024021aa048b3165742462 (127.24.235.65:42207)
I20260926 09:26:06.348865 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.65:35671
W20260926 09:26:06.430042 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:06.430264 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:06.430287 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:06.431493 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:06.431550 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:06.431588 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:06.432907 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:06.433156 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:06.433425 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:06.438616 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.438659 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.438690 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.438894 11817 server_base.cc:1034] running on GCE node
I20260926 09:26:06.439215 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:06.439462 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:06.440644 11817 hybrid_clock.cc:648] HybridClock initialized: now 1790414766440630 us; error 31 us; skew 500 ppm
I20260926 09:26:06.442174 11817 webserver.cc:460] Webserver started at http://127.24.235.66:40309/ using document root <none> and password file <none>
I20260926 09:26:06.442443 11817 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:06.442517 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:06.442670 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:06.443782 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "8f726ad60e264ce499ab0f7d00476960"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "a83a552607d753c1a4e376641adbfa91"
server_key_iv: "968867354e0ffcd557b3e3bc0a9cf364"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.444136 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "8f726ad60e264ce499ab0f7d00476960"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "a83a552607d753c1a4e376641adbfa91"
server_key_iv: "968867354e0ffcd557b3e3bc0a9cf364"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.445622 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:06.446527 11831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.446718 11817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:06.446830 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "8f726ad60e264ce499ab0f7d00476960"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "a83a552607d753c1a4e376641adbfa91"
server_key_iv: "968867354e0ffcd557b3e3bc0a9cf364"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.446924 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:06.480731 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:06.481110 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:06.481276 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:06.481531 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:06.481911 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:06.481968 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.482005 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:06.482054 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.487972 11817 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33227
I20260926 09:26:06.488021 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33227 every 8 connection(s)
I20260926 09:26:06.488385 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:06.491936 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11817
I20260926 09:26:06.492097 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:06.492319 25517 external_mini_cluster.cc:1428] Setting key 82107f0c2dfd79eb8ec95c4e30f1d0bb
I20260926 09:26:06.493278 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--builtin_ntp_servers=127.24.235.84:41807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:26:06.494143 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.489138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:59391 (local address 127.24.235.126:43217)
0926 09:26:06.489217 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.489220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.489548 (+ 328us) server_negotiation.cc:407] Connection header received
0926 09:26:06.490033 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.490034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.490048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.490127 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:06.490863 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.491373 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.492553 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.492822 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.492908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.493848 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:06.493849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:06.493849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:06.493854 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:06.493869 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:06.493874 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:06.493923 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.494034 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.494064 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 09:26:06.494370 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.489150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43217 (local address 127.24.235.66:59391)
0926 09:26:06.489370 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.489441 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.489946 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.490212 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.490216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.490224 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:06.490773 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.490777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.491460 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.491461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.492455 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.492458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.492920 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.492921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.493124 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.493687 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:06.493687 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:06.493716 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:06.493740 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:06.493933 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:06.493939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:06.493942 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.494026 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.494129 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260926 09:26:06.494863 11945 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43217
I20260926 09:26:06.494958 11945 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:06.495149 11945 heartbeater.cc:511] Master 127.24.235.126:43217 requested a full tablet report, sending...
I20260926 09:26:06.495538 11626 ts_manager.cc:194] Registered new tserver with Master: 8f726ad60e264ce499ab0f7d00476960 (127.24.235.66:33227)
I20260926 09:26:06.495913 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.66:59391
W20260926 09:26:06.576455 11948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:06.576681 11948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:06.576704 11948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:06.577907 11948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:06.577951 11948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:06.577977 11948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:06.579515 11948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:41807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:43217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:06.579771 11948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:06.580084 11948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:06.585402 11954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.585387 11956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:06.585381 11953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:06.585625 11948 server_base.cc:1034] running on GCE node
I20260926 09:26:06.585944 11948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:06.586208 11948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:06.587386 11948 hybrid_clock.cc:648] HybridClock initialized: now 1790414766587384 us; error 42 us; skew 500 ppm
I20260926 09:26:06.588879 11948 webserver.cc:460] Webserver started at http://127.24.235.67:40165/ using document root <none> and password file <none>
I20260926 09:26:06.589186 11948 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:06.589268 11948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:06.589411 11948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:06.590480 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "465a383038b6436e90d218e8b4c0fcde"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "22bc0e8283f1adafdd4bb36a8948d161"
server_key_iv: "0392237150fcb1e36fa08ebf17fa4c5a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.590876 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "465a383038b6436e90d218e8b4c0fcde"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "22bc0e8283f1adafdd4bb36a8948d161"
server_key_iv: "0392237150fcb1e36fa08ebf17fa4c5a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.592243 11948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:06.593051 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.593272 11948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:06.593372 11948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "465a383038b6436e90d218e8b4c0fcde"
format_stamp: "Formatted at 2026-09-26 09:26:06 on dist-test-slave-7zlw"
server_key: "22bc0e8283f1adafdd4bb36a8948d161"
server_key_iv: "0392237150fcb1e36fa08ebf17fa4c5a"
server_key_version: "encryptionkey@0"
I20260926 09:26:06.593463 11948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:06.612360 11948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:06.612705 11948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:06.612862 11948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:06.613164 11948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:06.613513 11948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:06.613569 11948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.613611 11948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:06.613646 11948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:06.619746 11948 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:33493
I20260926 09:26:06.619817 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:33493 every 8 connection(s)
I20260926 09:26:06.620193 11948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:06.625978 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.620985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:52709 (local address 127.24.235.126:43217)
0926 09:26:06.621070 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.621072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.621305 (+ 233us) server_negotiation.cc:407] Connection header received
0926 09:26:06.621852 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.621854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.621867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.621920 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:06.622681 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.623312 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.624319 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.624524 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.624605 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.625644 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:06.625645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:06.625645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:06.625651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:06.625667 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:06.625671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:06.625747 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.625856 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.625895 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 09:26:06.626222 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.620974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:43217 (local address 127.24.235.67:52709)
0926 09:26:06.621177 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.621212 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.621738 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.622007 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.622033 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.622043 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:06.622581 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.622586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.623413 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.623414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.624212 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.624214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.624616 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.624616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.624753 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.625408 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:06.625409 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:06.625438 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:06.625461 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:06.625748 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:06.625755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:06.625758 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.625843 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.625945 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260926 09:26:06.626731 12076 heartbeater.cc:347] Connected to a master server at 127.24.235.126:43217
I20260926 09:26:06.626812 12076 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:06.627012 12076 heartbeater.cc:511] Master 127.24.235.126:43217 requested a full tablet report, sending...
I20260926 09:26:06.627385 11626 ts_manager.cc:194] Registered new tserver with Master: 465a383038b6436e90d218e8b4c0fcde (127.24.235.67:33493)
I20260926 09:26:06.627799 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.67:52709
I20260926 09:26:06.629123 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 11948
I20260926 09:26:06.629231 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:06.629455 25517 external_mini_cluster.cc:1428] Setting key 089624a8a9db8785f7619940a362fb4b
I20260926 09:26:06.630373 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:06.634896 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.631805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.24.235.126:43217)
0926 09:26:06.631878 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.631881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.631953 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:26:06.632028 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.632029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.632040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.632113 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:06.632312 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.632776 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.633594 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.633789 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.633873 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.634599 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:06.634600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:06.634600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:06.634605 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:06.634631 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:06.634635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:06.634738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.634805 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.634814 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 09:26:06.636480 11626 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:06.638172 11626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:06.644555 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.24.235.67:33493)
0926 09:26:06.640778 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.640782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.640792 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:06.640832 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.640839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.640852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.640951 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:06.641152 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.641968 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.643434 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644112 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.644263 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.644386 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.644417 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.644457 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":87,"threads_started":1}
I20260926 09:26:06.644543 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:33493 (local address 127.0.0.1:46210)
0926 09:26:06.640611 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.640620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.640664 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.640968 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.640969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.640971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:06.641062 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.641064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.642062 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.642063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.643346 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.643349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644259 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.644260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.644321 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.644353 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.644400 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.644466 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":76,"threads_started":1}
I20260926 09:26:06.644613 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.24.235.66:33227)
0926 09:26:06.640759 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.640769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.640778 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:06.640812 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.640813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.640825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.640887 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:06.641134 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.642256 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.643372 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644037 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.644107 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.644200 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.644469 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.644521 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260926 09:26:06.644958 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42207 (local address 127.0.0.1:42872)
0926 09:26:06.640436 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.640475 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.640542 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.641303 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.641304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.641306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:06.641343 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.641345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.642586 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.642587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.643614 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.643617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644518 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.644519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.644575 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.644579 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.644817 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.644866 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260926 09:26:06.645342 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33227 (local address 127.0.0.1:36864)
0926 09:26:06.640646 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.640658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.640714 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.640930 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.640933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.640935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:06.641010 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.641045 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.642353 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.642354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.643276 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.643278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644237 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.644238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.644320 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.644335 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.644406 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.645294 (+ 888us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":848,"thread_start_us":120,"threads_started":1}
I20260926 09:26:06.645431 12010 tablet_service.cc:1467] Processing CreateTablet for tablet afa76d11ec30414585dd8bb28c67ff83 (DEFAULT_TABLE table=test-table [id=b59b44b98bc64e53ad7931bb6c02e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:06.645742 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.640437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.24.235.65:42207)
0926 09:26:06.641135 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.641139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.641149 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:06.641188 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.641190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.641204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.641426 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:06.641490 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.642505 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.643703 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.644428 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.645234 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.645333 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.645360 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.645660 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":88,"threads_started":1}
I20260926 09:26:06.645874 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa76d11ec30414585dd8bb28c67ff83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:06.646054 11879 tablet_service.cc:1467] Processing CreateTablet for tablet afa76d11ec30414585dd8bb28c67ff83 (DEFAULT_TABLE table=test-table [id=b59b44b98bc64e53ad7931bb6c02e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:06.646384 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa76d11ec30414585dd8bb28c67ff83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:06.648602 12090 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Bootstrap starting.
I20260926 09:26:06.649379 12090 tablet_bootstrap.cc:654] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:06.649374 11748 tablet_service.cc:1467] Processing CreateTablet for tablet afa76d11ec30414585dd8bb28c67ff83 (DEFAULT_TABLE table=test-table [id=b59b44b98bc64e53ad7931bb6c02e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:06.649708 12090 log.cc:824] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:06.649741 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa76d11ec30414585dd8bb28c67ff83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:06.649705 12091 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Bootstrap starting.
I20260926 09:26:06.650408 12091 tablet_bootstrap.cc:654] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:06.650583 12090 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: No bootstrap required, opened a new log
I20260926 09:26:06.650663 12090 ts_tablet_manager.cc:1397] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:26:06.650801 12091 log.cc:824] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:06.653388 12094 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: Bootstrap starting.
I20260926 09:26:06.653436 12090 raft_consensus.cc:348] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.653627 12090 raft_consensus.cc:374] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:06.653685 12091 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: No bootstrap required, opened a new log
I20260926 09:26:06.653707 12090 raft_consensus.cc:729] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465a383038b6436e90d218e8b4c0fcde, State: Initialized, Role: FOLLOWER
I20260926 09:26:06.653780 12091 ts_tablet_manager.cc:1397] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 09:26:06.653831 12090 consensus_queue.cc:260] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.654095 12090 ts_tablet_manager.cc:1428] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 09:26:06.654253 12076 heartbeater.cc:503] Master 127.24.235.126:43217 was elected leader, sending a full tablet report...
I20260926 09:26:06.654461 12094 tablet_bootstrap.cc:654] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:06.655258 12094 log.cc:824] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:06.656147 12094 tablet_bootstrap.cc:492] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: No bootstrap required, opened a new log
I20260926 09:26:06.656219 12094 ts_tablet_manager.cc:1397] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 09:26:06.656632 12091 raft_consensus.cc:348] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.656854 12091 raft_consensus.cc:374] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:06.656895 12091 raft_consensus.cc:729] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f726ad60e264ce499ab0f7d00476960, State: Initialized, Role: FOLLOWER
I20260926 09:26:06.657063 12091 consensus_queue.cc:260] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.657346 12091 ts_tablet_manager.cc:1428] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 09:26:06.657401 11945 heartbeater.cc:503] Master 127.24.235.126:43217 was elected leader, sending a full tablet report...
I20260926 09:26:06.658775 12094 raft_consensus.cc:348] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.658905 12094 raft_consensus.cc:374] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:06.658926 12094 raft_consensus.cc:729] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db557ea6e7024021aa048b3165742462, State: Initialized, Role: FOLLOWER
I20260926 09:26:06.659005 12094 consensus_queue.cc:260] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.659261 12094 ts_tablet_manager.cc:1428] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:06.659294 11814 heartbeater.cc:503] Master 127.24.235.126:43217 was elected leader, sending a full tablet report...
I20260926 09:26:06.728535 12097 raft_consensus.cc:482] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:06.728658 12097 raft_consensus.cc:504] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.729154 12097 leader_election.cc:288] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db557ea6e7024021aa048b3165742462 (127.24.235.65:42207), 465a383038b6436e90d218e8b4c0fcde (127.24.235.67:33493)
I20260926 09:26:06.732160 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.729327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42207 (local address 127.24.235.66:50153)
0926 09:26:06.729388 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.729401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.729452 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.729626 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.729627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.729629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:06.729707 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.729709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.730384 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.730385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.731259 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.731262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.731886 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.731887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.731957 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.731960 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.732043 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.732088 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:26:06.732160 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.729396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:50153 (local address 127.24.235.65:42207)
0926 09:26:06.729448 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.729450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.729462 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:26:06.729520 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.729521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.729534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.729619 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:06.729799 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.730295 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.731353 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.731805 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.731886 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.731894 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.732052 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.732089 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:26:06.732322 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.729486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:54573 (local address 127.24.235.67:33493)
0926 09:26:06.729534 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.729535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.729692 (+ 157us) server_negotiation.cc:407] Connection header received
0926 09:26:06.729729 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.729730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.729737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.729799 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:06.729953 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.730469 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.731481 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.731973 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.732054 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.732062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.732218 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.732258 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:26:06.732332 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.729350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:33493 (local address 127.24.235.66:54573)
0926 09:26:06.729550 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.729558 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:06.729632 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:06.729803 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:06.729804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:06.729806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:06.729882 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.729884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.730583 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.730584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.731397 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:06.731400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.732054 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.732055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:06.732141 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.732145 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:06.732211 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 09:26:06.732248 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":22,"thread_start_us":109,"threads_started":1}
I20260926 09:26:06.732534 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa76d11ec30414585dd8bb28c67ff83" candidate_uuid: "8f726ad60e264ce499ab0f7d00476960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db557ea6e7024021aa048b3165742462" is_pre_election: true
I20260926 09:26:06.732688 11768 raft_consensus.cc:2454] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f726ad60e264ce499ab0f7d00476960 in term 0.
I20260926 09:26:06.732942 11835 leader_election.cc:302] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f726ad60e264ce499ab0f7d00476960, db557ea6e7024021aa048b3165742462; no voters:
I20260926 09:26:06.733108 12097 raft_consensus.cc:2781] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:06.733172 12097 raft_consensus.cc:482] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:06.733093 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa76d11ec30414585dd8bb28c67ff83" candidate_uuid: "8f726ad60e264ce499ab0f7d00476960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465a383038b6436e90d218e8b4c0fcde" is_pre_election: true
I20260926 09:26:06.733188 12097 raft_consensus.cc:3037] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:06.733270 12030 raft_consensus.cc:2454] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f726ad60e264ce499ab0f7d00476960 in term 0.
I20260926 09:26:06.734012 12097 raft_consensus.cc:504] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.734165 12097 leader_election.cc:288] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [CANDIDATE]: Term 1 election: Requested vote from peers db557ea6e7024021aa048b3165742462 (127.24.235.65:42207), 465a383038b6436e90d218e8b4c0fcde (127.24.235.67:33493)
I20260926 09:26:06.734326 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa76d11ec30414585dd8bb28c67ff83" candidate_uuid: "8f726ad60e264ce499ab0f7d00476960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465a383038b6436e90d218e8b4c0fcde"
I20260926 09:26:06.734336 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa76d11ec30414585dd8bb28c67ff83" candidate_uuid: "8f726ad60e264ce499ab0f7d00476960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db557ea6e7024021aa048b3165742462"
I20260926 09:26:06.734412 12030 raft_consensus.cc:3037] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:06.734412 11768 raft_consensus.cc:3037] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:06.735356 11768 raft_consensus.cc:2454] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f726ad60e264ce499ab0f7d00476960 in term 1.
I20260926 09:26:06.735356 12030 raft_consensus.cc:2454] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f726ad60e264ce499ab0f7d00476960 in term 1.
I20260926 09:26:06.735540 11832 leader_election.cc:302] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465a383038b6436e90d218e8b4c0fcde, 8f726ad60e264ce499ab0f7d00476960; no voters:
I20260926 09:26:06.735666 12097 raft_consensus.cc:2781] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:06.735810 12097 raft_consensus.cc:686] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 1 LEADER]: Becoming Leader. State: Replica: 8f726ad60e264ce499ab0f7d00476960, State: Running, Role: LEADER
I20260926 09:26:06.735913 12097 consensus_queue.cc:237] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } }
I20260926 09:26:06.736847 11626 catalog_manager.cc:5495] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f726ad60e264ce499ab0f7d00476960 (127.24.235.66). New cstate: current_term: 1 leader_uuid: "8f726ad60e264ce499ab0f7d00476960" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f726ad60e264ce499ab0f7d00476960" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33227 } health_report { overall_health: HEALTHY } } }
W20260926 09:26:06.738958 11946 tablet.cc:2369] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:06.760782 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.758406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.24.235.66:33227)
0926 09:26:06.758474 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:06.758476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:06.758486 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:06.758527 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:06.758528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:06.758547 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:06.758678 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:06.758758 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.759233 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.759989 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:06.760195 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:06.760272 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:06.760425 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:06.760520 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:06.760643 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:06.760672 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 09:26:06.760683 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:06.762794 11768 raft_consensus.cc:1261] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 1 FOLLOWER]: Refusing update from remote peer 8f726ad60e264ce499ab0f7d00476960: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:06.762820 12030 raft_consensus.cc:1261] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 1 FOLLOWER]: Refusing update from remote peer 8f726ad60e264ce499ab0f7d00476960: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:06.763226 12097 consensus_queue.cc:1035] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db557ea6e7024021aa048b3165742462" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:06.763398 12100 consensus_queue.cc:1035] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "465a383038b6436e90d218e8b4c0fcde" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:06.769366 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:06.769455 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:06.770231 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87f32d1777ec43cfbc11c945c78799fe: Sending DeleteTablet for 3 replicas of tablet afa76d11ec30414585dd8bb28c67ff83
I20260926 09:26:06.770569 11748 tablet_service.cc:1514] Processing DeleteTablet for tablet afa76d11ec30414585dd8bb28c67ff83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:06 UTC) from {username='slave'} at 127.0.0.1:42872
I20260926 09:26:06.770586 12010 tablet_service.cc:1514] Processing DeleteTablet for tablet afa76d11ec30414585dd8bb28c67ff83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:06 UTC) from {username='slave'} at 127.0.0.1:46210
I20260926 09:26:06.770828 12113 tablet_replica.cc:330] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: stopping tablet replica
I20260926 09:26:06.770931 12113 raft_consensus.cc:2229] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:06.770938 11879 tablet_service.cc:1514] Processing DeleteTablet for tablet afa76d11ec30414585dd8bb28c67ff83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:06 UTC) from {username='slave'} at 127.0.0.1:36864
I20260926 09:26:06.771042 12113 raft_consensus.cc:2258] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:06.771071 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11686
I20260926 09:26:06.771224 12114 tablet_replica.cc:330] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462: stopping tablet replica
I20260926 09:26:06.771330 12114 raft_consensus.cc:2229] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:06.771425 12114 raft_consensus.cc:2258] T afa76d11ec30414585dd8bb28c67ff83 P db557ea6e7024021aa048b3165742462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:06.771476 12113 ts_tablet_manager.cc:1905] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:06.771663 12115 tablet_replica.cc:330] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: stopping tablet replica
I20260926 09:26:06.771749 12115 raft_consensus.cc:2229] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:06.771857 12115 raft_consensus.cc:2258] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:06.772336 12115 ts_tablet_manager.cc:1905] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:06.773870 12113 ts_tablet_manager.cc:1918] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:06.773939 12113 log.cc:1196] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/afa76d11ec30414585dd8bb28c67ff83
I20260926 09:26:06.774163 12113 ts_tablet_manager.cc:1939] T afa76d11ec30414585dd8bb28c67ff83 P 465a383038b6436e90d218e8b4c0fcde: Deleting consensus metadata
I20260926 09:26:06.774303 12115 ts_tablet_manager.cc:1918] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:06.774360 12115 log.cc:1196] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/afa76d11ec30414585dd8bb28c67ff83
I20260926 09:26:06.774570 11610 catalog_manager.cc:4841] TS 465a383038b6436e90d218e8b4c0fcde (127.24.235.67:33493): tablet afa76d11ec30414585dd8bb28c67ff83 (table test-table [id=b59b44b98bc64e53ad7931bb6c02e682]) successfully deleted
I20260926 09:26:06.774704 12115 ts_tablet_manager.cc:1939] T afa76d11ec30414585dd8bb28c67ff83 P 8f726ad60e264ce499ab0f7d00476960: Deleting consensus metadata
I20260926 09:26:06.775067 11612 catalog_manager.cc:4841] TS 8f726ad60e264ce499ab0f7d00476960 (127.24.235.66:33227): tablet afa76d11ec30414585dd8bb28c67ff83 (table test-table [id=b59b44b98bc64e53ad7931bb6c02e682]) successfully deleted
W20260926 09:26:06.780184 11613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:42207 (error 108)
I20260926 09:26:06.780683 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:06.780552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:42207 (local address 127.0.0.1:42876)
0926 09:26:06.780610 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:06.780655 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:42207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:26:06.780736 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11817
W20260926 09:26:06.785177 11613 catalog_manager.cc:4584] TS db557ea6e7024021aa048b3165742462 (127.24.235.65:42207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afa76d11ec30414585dd8bb28c67ff83: Network error: Client connection negotiation failed: client connection to 127.24.235.65:42207: connect: Connection refused (error 111)
I20260926 09:26:06.785454 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11948
I20260926 09:26:06.790915 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 11595
2026-09-26T09:26:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (778 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:06 dist-test-slave-7zlw krb5kdc[12119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:06 dist-test-slave-7zlw krb5kdc[12119](info): set up 2 sockets
Sep 26 09:26:06 dist-test-slave-7zlw krb5kdc[12119](info): commencing operation
krb5kdc: starting...
W20260926 09:26:07.837133 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:07 dist-test-slave-7zlw krb5kdc[12119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:07.963732 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36451
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:08.046957 12132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:08.047179 12132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:08.047204 12132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:08.048406 12132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:08.048451 12132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:08.048466 12132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:08.048478 12132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:08.048491 12132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:08.049983 12132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:08.050191 12132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:08.050369 12132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:08.055406 12140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.055457 12137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.055464 12138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.055626 12132 server_base.cc:1034] running on GCE node
I20260926 09:26:08.056011 12132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:08.056305 12132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:08.057469 12132 hybrid_clock.cc:648] HybridClock initialized: now 1790414768057452 us; error 48 us; skew 500 ppm
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:08.060111 12132 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:08.060642 12132 webserver.cc:460] Webserver started at http://127.24.235.126:40287/ using document root <none> and password file <none>
I20260926 09:26:08.060890 12132 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:08.060961 12132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:08.061146 12132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:08.062408 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "5705a18b53999b78d3e77f59ad10ab3f"
server_key_iv: "8eb40e6af1121d68879aa015c6311cea"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.062790 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "5705a18b53999b78d3e77f59ad10ab3f"
server_key_iv: "8eb40e6af1121d68879aa015c6311cea"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.064131 12132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:08.064868 12147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.065089 12132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:08.065189 12132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "5705a18b53999b78d3e77f59ad10ab3f"
server_key_iv: "8eb40e6af1121d68879aa015c6311cea"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.065280 12132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:08.081450 12132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:08.083547 12132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:08.083724 12132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:08.087984 12132 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36451
I20260926 09:26:08.088019 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36451 every 8 connection(s)
I20260926 09:26:08.088398 12132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:08.089088 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12132
I20260926 09:26:08.089222 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:08.089443 25517 external_mini_cluster.cc:1428] Setting key 7d2f8ba179b3b152f9cd5573873a8115
I20260926 09:26:08.089480 12200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:08.092836 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: Bootstrap starting.
I20260926 09:26:08.093583 12200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:08.093894 12200 log.cc:824] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:08.094638 12200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: No bootstrap required, opened a new log
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:08.097257 12200 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } }
I20260926 09:26:08.097453 12200 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:08.097499 12200 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cdda53a5d0446c9b4ce87a56c7d35f, State: Initialized, Role: FOLLOWER
I20260926 09:26:08.097642 12200 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } }
I20260926 09:26:08.097743 12200 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:08.097769 12200 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:08.097838 12200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:08.098570 12200 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } }
I20260926 09:26:08.098680 12200 leader_election.cc:302] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8cdda53a5d0446c9b4ce87a56c7d35f; no voters:
I20260926 09:26:08.098846 12200 leader_election.cc:288] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:08.098914 12205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:08.099081 12205 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [term 1 LEADER]: Becoming Leader. State: Replica: f8cdda53a5d0446c9b4ce87a56c7d35f, State: Running, Role: LEADER
I20260926 09:26:08.099138 12200 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:08.099170 12205 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } }
I20260926 09:26:08.099594 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } } }
I20260926 09:26:08.099710 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:08.099578 12207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8cdda53a5d0446c9b4ce87a56c7d35f. Latest consensus state: current_term: 1 leader_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cdda53a5d0446c9b4ce87a56c7d35f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36451 } } }
I20260926 09:26:08.099953 12214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:08.099963 12207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:08.100755 12214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:08.101704 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.089962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.24.235.126:36451)
0926 09:26:08.090162 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.090169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.090196 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:26:08.090627 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.090634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.090681 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.090824 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:08.091648 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.092558 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.093684 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.094167 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.094253 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.096850 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:08.096856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:08.096856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:08.096872 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:08.098891 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.099342 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.099346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.099346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.099445 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.100039 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.100040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.100040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.100492 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:08.101289 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.101363 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.101503 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260926 09:26:08.102820 12214 catalog_manager.cc:1347] Generated new cluster ID: 951dfa2e42f24616aeb91f5663b600e9
I20260926 09:26:08.102875 12214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:08.118139 12214 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:08.118880 12214 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:08.128051 12214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: Generated new TSK 0
I20260926 09:26:08.128305 12214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
Entry for principal kudu/127.24.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab.
I20260926 09:26:08.156970 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:08.242668 12226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:08.242877 12226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:08.242901 12226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:08.244019 12226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:08.244060 12226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:08.244087 12226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:08.245628 12226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.65.keytab
--principal=kudu/127.24.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:08.245834 12226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:08.246054 12226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:08.251224 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.251250 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.251276 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.251327 12226 server_base.cc:1034] running on GCE node
I20260926 09:26:08.251706 12226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:08.251972 12226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:08.253157 12226 hybrid_clock.cc:648] HybridClock initialized: now 1790414768253115 us; error 47 us; skew 500 ppm
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:08.255996 12226 init.cc:377] Logged in from keytab as kudu/127.24.235.65@KRBTEST.COM (short username kudu)
I20260926 09:26:08.256493 12226 webserver.cc:460] Webserver started at http://127.24.235.65:41141/ using document root <none> and password file <none>
I20260926 09:26:08.256759 12226 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:08.256807 12226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:08.256982 12226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:08.258162 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "8346970ac9e44d418041541a62d256bc"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "fdde3ab00608d656aea12ab2e3a2be1d"
server_key_iv: "6487ae4552a97150fa125f238bc08bf9"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.258476 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "8346970ac9e44d418041541a62d256bc"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "fdde3ab00608d656aea12ab2e3a2be1d"
server_key_iv: "6487ae4552a97150fa125f238bc08bf9"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.259857 12226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:08.260665 12241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.260883 12226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:08.260988 12226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "8346970ac9e44d418041541a62d256bc"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "fdde3ab00608d656aea12ab2e3a2be1d"
server_key_iv: "6487ae4552a97150fa125f238bc08bf9"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.261132 12226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:08.280316 12226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:08.282483 12226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:08.282662 12226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:08.282922 12226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:08.283248 12226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:08.283311 12226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.283353 12226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:08.283384 12226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.289440 12226 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:45775
I20260926 09:26:08.289546 12354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:45775 every 8 connection(s)
I20260926 09:26:08.289878 12226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:26:08.293071 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12226
I20260926 09:26:08.293215 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:08.293426 25517 external_mini_cluster.cc:1428] Setting key d7f4109a2c22fc7c848b0098c9889437
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.65@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:08.301569 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.290788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:59973 (local address 127.24.235.126:36451)
0926 09:26:08.290859 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.290862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.291102 (+ 240us) server_negotiation.cc:407] Connection header received
0926 09:26:08.291633 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.291633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.291643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.291704 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:08.292747 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.293473 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.294735 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.294962 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.295037 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.297777 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:08.297779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:08.297779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:08.297788 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:08.299744 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.300322 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.300323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.300324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.300392 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.300769 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.300770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.300771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.300935 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:08.301032 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.301345 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.301448 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 09:26:08.301601 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.290768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36451 (local address 127.24.235.65:59973)
0926 09:26:08.290963 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.291010 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.291546 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.291765 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.291768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.292409 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:08.292624 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.292629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.293589 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.293590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.294617 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.294620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.295063 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.295065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.295199 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.295837 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:08.295850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:08.297675 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:08.299863 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.299870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.299872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.300190 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.300464 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.300464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.300464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.300651 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.301008 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:08.301045 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:08.301180 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.301293 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.301426 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260926 09:26:08.302248 12355 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36451
I20260926 09:26:08.302338 12355 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:08.302534 12355 heartbeater.cc:511] Master 127.24.235.126:36451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.235.66@KRBTEST.COM; defaulting to no policy
I20260926 09:26:08.303125 12164 ts_manager.cc:194] Registered new tserver with Master: 8346970ac9e44d418041541a62d256bc (127.24.235.65:45775)
I20260926 09:26:08.303872 12164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.65@KRBTEST.COM'} at 127.24.235.65:59973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
Entry for principal kudu/127.24.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab.
I20260926 09:26:08.323938 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:08.407719 12360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:08.407953 12360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:08.407979 12360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:08.409348 12360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:08.409397 12360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:08.409431 12360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:08.411052 12360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.66.keytab
--principal=kudu/127.24.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:08.411339 12360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:08.411589 12360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:08.416841 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.416864 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.416924 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.417084 12360 server_base.cc:1034] running on GCE node
I20260926 09:26:08.417323 12360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:08.417544 12360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:08.418686 12360 hybrid_clock.cc:648] HybridClock initialized: now 1790414768418676 us; error 32 us; skew 500 ppm
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:08.421235 12360 init.cc:377] Logged in from keytab as kudu/127.24.235.66@KRBTEST.COM (short username kudu)
I20260926 09:26:08.421744 12360 webserver.cc:460] Webserver started at http://127.24.235.66:38179/ using document root <none> and password file <none>
I20260926 09:26:08.421977 12360 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:08.422046 12360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:08.422200 12360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:08.423305 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "6c6659b1c8454526b4920ab02b8a2b46"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "35d04bcf2525a364581adc60b195fe2a"
server_key_iv: "672e0f2d8df58ff0aa3d17ce983004c0"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.423687 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "6c6659b1c8454526b4920ab02b8a2b46"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "35d04bcf2525a364581adc60b195fe2a"
server_key_iv: "672e0f2d8df58ff0aa3d17ce983004c0"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.425268 12360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:08.426087 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.426276 12360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:08.426379 12360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "6c6659b1c8454526b4920ab02b8a2b46"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "35d04bcf2525a364581adc60b195fe2a"
server_key_iv: "672e0f2d8df58ff0aa3d17ce983004c0"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.426466 12360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:08.438884 12360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:08.441164 12360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:08.441345 12360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:08.441601 12360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:08.441929 12360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:08.441987 12360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.442022 12360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:08.442065 12360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.448690 12360 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:35305
I20260926 09:26:08.448804 12488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:35305 every 8 connection(s)
I20260926 09:26:08.449208 12360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:08.449400 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12360
I20260926 09:26:08.449520 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:08.449738 25517 external_mini_cluster.cc:1428] Setting key 1ffa61e50f0f894e7230f64a9bbfd400
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.66@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.235.67@KRBTEST.COM; defaulting to no policy
I20260926 09:26:08.460213 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.450054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:34637 (local address 127.24.235.126:36451)
0926 09:26:08.450109 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.450111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.450498 (+ 387us) server_negotiation.cc:407] Connection header received
0926 09:26:08.451343 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.451344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.451355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.451410 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:08.452711 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.453541 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.454616 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.454936 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.455015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.457299 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:08.457300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:08.457301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:08.457310 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:08.458877 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.459319 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.459320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.459320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.459371 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.459588 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.459588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.459589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.459687 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:08.459739 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.460068 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.460143 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:26:08.460472 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.450043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36451 (local address 127.24.235.66:34637)
0926 09:26:08.450396 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.450434 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.451208 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.451503 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.451508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.452399 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:08.452605 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.452610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.453631 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.453632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.454523 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.454527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.455028 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.455029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.455169 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.455759 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:08.455767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:08.457206 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:08.458984 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.458991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.458993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.459242 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.459435 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.459436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.459436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.459530 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.459755 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:08.459760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:08.459873 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.459995 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.460198 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":98,"threads_started":1}
I20260926 09:26:08.461001 12489 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36451
I20260926 09:26:08.461148 12489 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:08.461337 12489 heartbeater.cc:511] Master 127.24.235.126:36451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.67@KRBTEST.COM" created.
I20260926 09:26:08.461786 12164 ts_manager.cc:194] Registered new tserver with Master: 6c6659b1c8454526b4920ab02b8a2b46 (127.24.235.66:35305)
I20260926 09:26:08.462312 12164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.66@KRBTEST.COM'} at 127.24.235.66:34637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
Entry for principal kudu/127.24.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab.
I20260926 09:26:08.479559 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:08.562995 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:08.563338 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:08.563383 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:08.564689 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:08.564759 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:08.564807 12494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:08.566485 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.67.keytab
--principal=kudu/127.24.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:08.566777 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:08.567052 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:08.572779 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.572873 12494 server_base.cc:1034] running on GCE node
W20260926 09:26:08.572753 12500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.572762 12499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.573293 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:08.573556 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:08.574726 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790414768574708 us; error 38 us; skew 500 ppm
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:08.577256 12494 init.cc:377] Logged in from keytab as kudu/127.24.235.67@KRBTEST.COM (short username kudu)
I20260926 09:26:08.577661 12494 webserver.cc:460] Webserver started at http://127.24.235.67:45161/ using document root <none> and password file <none>
I20260926 09:26:08.577859 12494 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:08.577899 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:08.577987 12494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:08.579039 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "f639f05e80fb4356a5b7e00956ab730d"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "d4b1731e497901c0e1ad51e7b6185833"
server_key_iv: "60b7900e62b32b69367b4bb11dcf705a"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.579397 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "f639f05e80fb4356a5b7e00956ab730d"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "d4b1731e497901c0e1ad51e7b6185833"
server_key_iv: "60b7900e62b32b69367b4bb11dcf705a"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.580725 12494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:08.581466 12509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.581647 12494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.581753 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "f639f05e80fb4356a5b7e00956ab730d"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "d4b1731e497901c0e1ad51e7b6185833"
server_key_iv: "60b7900e62b32b69367b4bb11dcf705a"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.581827 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:08.592981 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:08.595189 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:08.595366 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:08.595613 12494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:08.595947 12494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:08.596002 12494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.596035 12494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:08.596082 12494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.602586 12494 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:39067
I20260926 09:26:08.602629 12622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:39067 every 8 connection(s)
I20260926 09:26:08.602959 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:08.604863 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12494
I20260926 09:26:08.604979 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:08.605248 25517 external_mini_cluster.cc:1428] Setting key fe9b593463532beacb877bcd9c327219
Sep 26 09:26:08 dist-test-slave-7zlw krb5kdc[12119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414768, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.67@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:08.612694 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.603688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:56963 (local address 127.24.235.126:36451)
0926 09:26:08.603732 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.603734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.603934 (+ 200us) server_negotiation.cc:407] Connection header received
0926 09:26:08.604438 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.604440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.604458 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.604517 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:08.605662 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.606259 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.607310 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.607524 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.607603 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.609956 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:08.609957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:08.609958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:08.609966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:08.611266 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.611671 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.611673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.611673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.611736 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.612035 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.612036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.612036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.612151 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:08.612228 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.612517 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.612600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:08.612888 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.603678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36451 (local address 127.24.235.67:56963)
0926 09:26:08.603815 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.603850 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.604327 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.604569 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.604572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.605284 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:08.605582 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.605589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.606362 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.606363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.607196 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.607198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.607611 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.607612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.607754 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.608269 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:08.608278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:08.609858 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:08.611352 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.611360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.611362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.611592 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.611834 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:08.611835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:08.611836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:08.611950 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:08.612217 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:08.612224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:08.612381 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.612494 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.612658 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260926 09:26:08.613430 12623 heartbeater.cc:347] Connected to a master server at 127.24.235.126:36451
I20260926 09:26:08.613521 12623 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:08.613731 12623 heartbeater.cc:511] Master 127.24.235.126:36451 requested a full tablet report, sending...
I20260926 09:26:08.614125 12164 ts_manager.cc:194] Registered new tserver with Master: f639f05e80fb4356a5b7e00956ab730d (127.24.235.67:39067)
I20260926 09:26:08.614498 12164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.235.67@KRBTEST.COM'} at 127.24.235.67:56963
I20260926 09:26:08.617067 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:08.624420 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.618423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.24.235.126:36451)
0926 09:26:08.618506 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.618509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.618620 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:26:08.618704 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.618705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.618717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.618780 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:08.619324 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.619805 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.620531 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.620740 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.620823 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.621988 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:08.621990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:08.621990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:08.621997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:08.623219 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.623563 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.623564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.623565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.623619 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:08.623825 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:08.623826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:08.623826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:08.623979 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:08.624084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.624239 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.624318 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 09:26:08.626204 12164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:08.627786 12164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:08.634711 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:39067 (local address 127.0.0.1:36048)
0926 09:26:08.630714 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.630726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.630798 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.631122 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.631123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.631126 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:08.631187 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.631188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632191 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.632192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.633588 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.633589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634381 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.634461 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634474 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.634578 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.634622 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260926 09:26:08.634719 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.24.235.67:39067)
0926 09:26:08.630948 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.630953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.630962 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:08.631003 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.631005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.631022 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.631257 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:08.631342 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632090 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.633689 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634290 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634381 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.634597 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.634636 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":90,"threads_started":1}
I20260926 09:26:08.634797 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.24.235.65:45775)
0926 09:26:08.630833 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.630839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.631257 (+ 418us) server_negotiation.cc:407] Connection header received
0926 09:26:08.631299 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.631301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.631316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.631388 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:08.631515 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632530 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.633570 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634311 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634389 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634449 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.634666 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.634706 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260926 09:26:08.634974 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:45775 (local address 127.0.0.1:34544)
0926 09:26:08.631128 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.631138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.631186 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.631407 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.631408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.631411 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:08.631451 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.631453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632628 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.632629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.633474 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.633477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634477 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.634531 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634533 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.634578 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.634607 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":90,"threads_started":1}
I20260926 09:26:08.635295 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35305 (local address 127.0.0.1:46592)
0926 09:26:08.630482 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.630525 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.630595 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.631049 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.631052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.631055 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:08.631126 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.631128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632455 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.632456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.633587 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.633589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634426 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.634492 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634497 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.634527 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.634594 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":48,"threads_started":1}
I20260926 09:26:08.635473 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.630673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.24.235.66:35305)
0926 09:26:08.630885 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.630889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.630898 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:08.630940 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.630942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.630958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.631032 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:08.631213 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.632335 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.633665 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.634334 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.634673 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.634748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.635376 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.635414 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260926 09:26:08.635522 12557 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01349cfc564dc48da71efc6e420aae (DEFAULT_TABLE table=test-table [id=92dcecd559944d76a583dc8d63d9b6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:08.635602 12289 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01349cfc564dc48da71efc6e420aae (DEFAULT_TABLE table=test-table [id=92dcecd559944d76a583dc8d63d9b6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:08.635898 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01349cfc564dc48da71efc6e420aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:08.635898 12289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01349cfc564dc48da71efc6e420aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:08.637894 12423 tablet_service.cc:1467] Processing CreateTablet for tablet 8e01349cfc564dc48da71efc6e420aae (DEFAULT_TABLE table=test-table [id=92dcecd559944d76a583dc8d63d9b6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:08.638224 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e01349cfc564dc48da71efc6e420aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:08.639496 12641 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Bootstrap starting.
I20260926 09:26:08.640266 12641 tablet_bootstrap.cc:654] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:08.640875 12641 log.cc:824] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:08.641206 12640 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Bootstrap starting.
I20260926 09:26:08.641678 12641 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: No bootstrap required, opened a new log
I20260926 09:26:08.641772 12641 ts_tablet_manager.cc:1397] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:26:08.641873 12640 tablet_bootstrap.cc:654] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:08.642364 12640 log.cc:824] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:08.643064 12640 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: No bootstrap required, opened a new log
I20260926 09:26:08.643114 12640 ts_tablet_manager.cc:1397] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:26:08.643575 12641 raft_consensus.cc:348] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.643790 12641 raft_consensus.cc:374] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:08.643836 12641 raft_consensus.cc:729] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f639f05e80fb4356a5b7e00956ab730d, State: Initialized, Role: FOLLOWER
I20260926 09:26:08.643970 12641 consensus_queue.cc:260] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.644289 12641 ts_tablet_manager.cc:1428] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 09:26:08.644325 12623 heartbeater.cc:503] Master 127.24.235.126:36451 was elected leader, sending a full tablet report...
I20260926 09:26:08.641319 12643 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Bootstrap starting.
I20260926 09:26:08.644928 12643 tablet_bootstrap.cc:654] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:08.645259 12640 raft_consensus.cc:348] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.645417 12643 log.cc:824] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:08.645455 12640 raft_consensus.cc:374] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:08.645510 12640 raft_consensus.cc:729] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8346970ac9e44d418041541a62d256bc, State: Initialized, Role: FOLLOWER
I20260926 09:26:08.645627 12640 consensus_queue.cc:260] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.645932 12640 ts_tablet_manager.cc:1428] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:08.645947 12355 heartbeater.cc:503] Master 127.24.235.126:36451 was elected leader, sending a full tablet report...
I20260926 09:26:08.646176 12643 tablet_bootstrap.cc:492] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: No bootstrap required, opened a new log
I20260926 09:26:08.646230 12643 ts_tablet_manager.cc:1397] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 09:26:08.647749 12643 raft_consensus.cc:348] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.647938 12643 raft_consensus.cc:374] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:08.647989 12643 raft_consensus.cc:729] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6659b1c8454526b4920ab02b8a2b46, State: Initialized, Role: FOLLOWER
I20260926 09:26:08.648115 12643 consensus_queue.cc:260] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.648344 12643 ts_tablet_manager.cc:1428] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 09:26:08.648414 12489 heartbeater.cc:503] Master 127.24.235.126:36451 was elected leader, sending a full tablet report...
W20260926 09:26:08.700031 12490 tablet.cc:2369] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:08.790717 12647 raft_consensus.cc:482] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 09:26:08.790719 12356 tablet.cc:2369] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:08.790951 12647 raft_consensus.cc:504] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.791353 12647 leader_election.cc:288] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c6659b1c8454526b4920ab02b8a2b46 (127.24.235.66:35305), f639f05e80fb4356a5b7e00956ab730d (127.24.235.67:39067)
I20260926 09:26:08.794277 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.791579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35305 (local address 127.24.235.65:35937)
0926 09:26:08.791638 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.791651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.791710 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.791869 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.791870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.791871 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:08.791955 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.791957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.792620 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.792621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.793413 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.793415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.793988 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.793989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.794072 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.794075 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.794150 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.794203 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:26:08.794324 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.791635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:35937 (local address 127.24.235.66:35305)
0926 09:26:08.791681 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.791683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.791694 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:26:08.791790 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.791791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.791799 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.791865 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:08.792030 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.792528 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.793525 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.793904 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.793989 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.794000 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.794157 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.794252 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:26:08.794409 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.791670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:33637 (local address 127.24.235.67:39067)
0926 09:26:08.791718 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.791721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.791830 (+ 109us) server_negotiation.cc:407] Connection header received
0926 09:26:08.791907 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.791908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.791917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.791989 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:08.792161 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.792615 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.793634 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.794021 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.794105 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.794115 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.794298 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.794339 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:26:08.794409 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.791596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:39067 (local address 127.24.235.65:33637)
0926 09:26:08.791745 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.791754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:08.791838 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:08.791992 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:08.791993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:08.791994 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:08.792072 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.792073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.792717 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.792718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.793540 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:08.793543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.794130 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.794131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:08.794197 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.794200 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:08.794285 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 09:26:08.794325 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260926 09:26:08.794651 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01349cfc564dc48da71efc6e420aae" candidate_uuid: "8346970ac9e44d418041541a62d256bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6659b1c8454526b4920ab02b8a2b46" is_pre_election: true
I20260926 09:26:08.794705 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01349cfc564dc48da71efc6e420aae" candidate_uuid: "8346970ac9e44d418041541a62d256bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f639f05e80fb4356a5b7e00956ab730d" is_pre_election: true
I20260926 09:26:08.794849 12577 raft_consensus.cc:2454] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8346970ac9e44d418041541a62d256bc in term 0.
I20260926 09:26:08.794849 12443 raft_consensus.cc:2454] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8346970ac9e44d418041541a62d256bc in term 0.
I20260926 09:26:08.795068 12243 leader_election.cc:302] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8346970ac9e44d418041541a62d256bc, f639f05e80fb4356a5b7e00956ab730d; no voters:
I20260926 09:26:08.795195 12647 raft_consensus.cc:2781] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:08.795253 12647 raft_consensus.cc:482] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:08.795282 12647 raft_consensus.cc:3037] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:08.795995 12647 raft_consensus.cc:504] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.796149 12647 leader_election.cc:288] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [CANDIDATE]: Term 1 election: Requested vote from peers 6c6659b1c8454526b4920ab02b8a2b46 (127.24.235.66:35305), f639f05e80fb4356a5b7e00956ab730d (127.24.235.67:39067)
I20260926 09:26:08.796332 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01349cfc564dc48da71efc6e420aae" candidate_uuid: "8346970ac9e44d418041541a62d256bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6659b1c8454526b4920ab02b8a2b46"
I20260926 09:26:08.796401 12443 raft_consensus.cc:3037] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:08.796371 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e01349cfc564dc48da71efc6e420aae" candidate_uuid: "8346970ac9e44d418041541a62d256bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f639f05e80fb4356a5b7e00956ab730d"
I20260926 09:26:08.796504 12577 raft_consensus.cc:3037] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:08.797318 12443 raft_consensus.cc:2454] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8346970ac9e44d418041541a62d256bc in term 1.
I20260926 09:26:08.797446 12577 raft_consensus.cc:2454] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8346970ac9e44d418041541a62d256bc in term 1.
I20260926 09:26:08.797506 12245 leader_election.cc:302] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6659b1c8454526b4920ab02b8a2b46, 8346970ac9e44d418041541a62d256bc; no voters:
I20260926 09:26:08.797581 12647 raft_consensus.cc:2781] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:08.797778 12647 raft_consensus.cc:686] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 1 LEADER]: Becoming Leader. State: Replica: 8346970ac9e44d418041541a62d256bc, State: Running, Role: LEADER
I20260926 09:26:08.797927 12647 consensus_queue.cc:237] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } }
I20260926 09:26:08.798835 12164 catalog_manager.cc:5495] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8346970ac9e44d418041541a62d256bc (127.24.235.65). New cstate: current_term: 1 leader_uuid: "8346970ac9e44d418041541a62d256bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8346970ac9e44d418041541a62d256bc" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 45775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 } health_report { overall_health: UNKNOWN } } }
I20260926 09:26:08.818141 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.815516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.24.235.65:45775)
0926 09:26:08.815581 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.815583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:08.815593 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:08.815640 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:08.815641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:08.815663 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:08.815745 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:08.815899 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.816396 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.817261 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:08.817523 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:08.817607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:08.817799 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:08.817918 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:08.818008 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:08.818032 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 09:26:08.818052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:26:08.820149 12443 raft_consensus.cc:1261] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 1 FOLLOWER]: Refusing update from remote peer 8346970ac9e44d418041541a62d256bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:08.820149 12577 raft_consensus.cc:1261] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 1 FOLLOWER]: Refusing update from remote peer 8346970ac9e44d418041541a62d256bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:08.820533 12647 consensus_queue.cc:1035] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6659b1c8454526b4920ab02b8a2b46" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:08.820783 12650 consensus_queue.cc:1035] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f639f05e80fb4356a5b7e00956ab730d" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:08.823328 12654 mvcc.cc:205] Tried to move back new op lower bound from 7333538893084258304 to 7333538892997181440. Current Snapshot: MvccSnapshot[applied={T|T < 7333538893084258304}]
I20260926 09:26:08.823792 12655 mvcc.cc:205] Tried to move back new op lower bound from 7333538893084258304 to 7333538892997181440. Current Snapshot: MvccSnapshot[applied={T|T < 7333538893084258304}]
I20260926 09:26:08.825426 12164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:08.825510 12164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:08.826225 12164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8cdda53a5d0446c9b4ce87a56c7d35f: Sending DeleteTablet for 3 replicas of tablet 8e01349cfc564dc48da71efc6e420aae
I20260926 09:26:08.826568 12289 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01349cfc564dc48da71efc6e420aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:08 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:34544
I20260926 09:26:08.826602 12557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01349cfc564dc48da71efc6e420aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:08 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:36048
I20260926 09:26:08.826645 12423 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e01349cfc564dc48da71efc6e420aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:08 UTC) from {username='kudu', principal='kudu/127.24.235.126@KRBTEST.COM'} at 127.0.0.1:46592
I20260926 09:26:08.826844 12665 tablet_replica.cc:330] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46: stopping tablet replica
I20260926 09:26:08.826951 12665 raft_consensus.cc:2229] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:08.826972 12664 tablet_replica.cc:330] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: stopping tablet replica
I20260926 09:26:08.827087 12663 tablet_replica.cc:330] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc: stopping tablet replica
I20260926 09:26:08.827172 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12226
I20260926 09:26:08.827185 12663 raft_consensus.cc:2229] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:08.827191 12664 raft_consensus.cc:2229] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:08.827291 12663 raft_consensus.cc:2258] T 8e01349cfc564dc48da71efc6e420aae P 8346970ac9e44d418041541a62d256bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:08.827813 12664 raft_consensus.cc:2258] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:08.828342 12664 ts_tablet_manager.cc:1905] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:08.830541 12664 ts_tablet_manager.cc:1918] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:08.830631 12664 log.cc:1196] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/8e01349cfc564dc48da71efc6e420aae
I20260926 09:26:08.830926 12664 ts_tablet_manager.cc:1939] T 8e01349cfc564dc48da71efc6e420aae P f639f05e80fb4356a5b7e00956ab730d: Deleting consensus metadata
I20260926 09:26:08.831367 12149 catalog_manager.cc:4841] TS f639f05e80fb4356a5b7e00956ab730d (127.24.235.67:39067): tablet 8e01349cfc564dc48da71efc6e420aae (table test-table [id=92dcecd559944d76a583dc8d63d9b6c7]) successfully deleted
I20260926 09:26:08.833289 12665 raft_consensus.cc:2258] T 8e01349cfc564dc48da71efc6e420aae P 6c6659b1c8454526b4920ab02b8a2b46 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 09:26:08.833384 12148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:45775 (error 108)
I20260926 09:26:08.833703 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12360
I20260926 09:26:08.834322 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.833794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:45775 (local address 127.0.0.1:34554)
0926 09:26:08.834237 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.834280 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:45775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
W20260926 09:26:08.837188 12148 catalog_manager.cc:4584] TS 8346970ac9e44d418041541a62d256bc (127.24.235.65:45775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e01349cfc564dc48da71efc6e420aae: Network error: Client connection negotiation failed: client connection to 127.24.235.65:45775: connect: Connection refused (error 111)
I20260926 09:26:08.839161 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12494
I20260926 09:26:08.839339 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.839156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:35305 (local address 127.0.0.1:46596)
0926 09:26:08.839233 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:08.839295 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.66:35305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 09:26:08.844177 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12132
2026-09-26T09:26:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T09:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:08Z Disabled control of system clock
I20260926 09:26:08.873245 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33031
--webserver_interface=127.24.235.126
--webserver_port=0
--builtin_ntp_servers=127.24.235.84:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:08.954969 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:08.955209 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:08.955231 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:08.956362 12670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:08.956403 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:08.956415 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:08.956429 12670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:08.956441 12670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:08.957868 12670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:08.958094 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:08.958348 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:08.963619 12678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.963642 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:08.963586 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:08.964001 12670 server_base.cc:1034] running on GCE node
I20260926 09:26:08.964264 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:08.964551 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:08.965732 12670 hybrid_clock.cc:648] HybridClock initialized: now 1790414768965716 us; error 27 us; skew 500 ppm
I20260926 09:26:08.967219 12670 webserver.cc:460] Webserver started at http://127.24.235.126:33175/ using document root <none> and password file <none>
I20260926 09:26:08.967479 12670 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:08.967526 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:08.967622 12670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:08.968753 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "3489ed6e6d0240bdabd3651d9a0b606f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "f10271b72a66c3056ac6833549e0ba7c"
server_key_iv: "60dc126bbb5e9789c79ac596bce84f60"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.969185 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "3489ed6e6d0240bdabd3651d9a0b606f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "f10271b72a66c3056ac6833549e0ba7c"
server_key_iv: "60dc126bbb5e9789c79ac596bce84f60"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.970497 12670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:08.971386 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:08.971653 12670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:08.971763 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "3489ed6e6d0240bdabd3651d9a0b606f"
format_stamp: "Formatted at 2026-09-26 09:26:08 on dist-test-slave-7zlw"
server_key: "f10271b72a66c3056ac6833549e0ba7c"
server_key_iv: "60dc126bbb5e9789c79ac596bce84f60"
server_key_version: "encryptionkey@0"
I20260926 09:26:08.971849 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:08.982992 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:08.983392 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:08.983583 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:08.988310 12670 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33031
I20260926 09:26:08.988451 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33031 every 8 connection(s)
I20260926 09:26:08.988811 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:08.989884 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:08.992259 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: Bootstrap starting.
I20260926 09:26:08.992887 12737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:08.993223 12737 log.cc:824] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:08.993923 12737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: No bootstrap required, opened a new log
I20260926 09:26:08.995661 12737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } }
I20260926 09:26:08.995803 12737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:08.995826 12737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489ed6e6d0240bdabd3651d9a0b606f, State: Initialized, Role: FOLLOWER
I20260926 09:26:08.996007 12737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } }
I20260926 09:26:08.996070 12737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:08.996093 12737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:08.996160 12737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:08.996685 12737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } }
I20260926 09:26:08.996922 12737 leader_election.cc:302] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3489ed6e6d0240bdabd3651d9a0b606f; no voters:
I20260926 09:26:08.997148 12737 leader_election.cc:288] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:08.997198 12740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:08.997326 12740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [term 1 LEADER]: Becoming Leader. State: Replica: 3489ed6e6d0240bdabd3651d9a0b606f, State: Running, Role: LEADER
I20260926 09:26:08.997437 12737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:08.997416 12740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } }
I20260926 09:26:08.997723 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3489ed6e6d0240bdabd3651d9a0b606f. Latest consensus state: current_term: 1 leader_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } } }
I20260926 09:26:08.997793 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:08.998003 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12670
I20260926 09:26:08.998159 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:08.998140 12745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:08.998437 25517 external_mini_cluster.cc:1428] Setting key db285b9d004ce92f40eca91f63ca9056
I20260926 09:26:08.998677 12745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:08.999943 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3489ed6e6d0240bdabd3651d9a0b606f" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33031 } } }
I20260926 09:26:09.000059 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:09.001880 12745 catalog_manager.cc:1347] Generated new cluster ID: 1e0eaa83441040a3949ff419935efb2c
I20260926 09:26:09.001930 12745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:09.006313 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:08.999101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.24.235.126:33031)
0926 09:26:08.999465 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:08.999476 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.000022 (+ 546us) server_negotiation.cc:407] Connection header received
0926 09:26:09.000240 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.000248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.000328 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.000440 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:09.000823 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.002603 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.003811 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.004435 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.004534 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.005517 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:09.005521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:09.005521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:09.005743 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:09.005785 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:09.005799 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:09.005868 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.005991 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.006079 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":104,"threads_started":1}
I20260926 09:26:09.026343 12745 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:09.027096 12745 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:09.032701 12745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: Generated new TSK 0
I20260926 09:26:09.032954 12745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:09.038362 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.65:0
--local_ip_for_outbound_sockets=127.24.235.65
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--builtin_ntp_servers=127.24.235.84:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:09.121973 12761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:09.122233 12761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:09.122272 12761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:09.123534 12761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:09.123594 12761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:09.123637 12761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.65
I20260926 09:26:09.124997 12761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.235.65
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.65
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:09.125325 12761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:09.125640 12761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:09.131047 12769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:09.131043 12761 server_base.cc:1034] running on GCE node
W20260926 09:26:09.131078 12767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:09.131079 12766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:09.131562 12761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:09.131809 12761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:09.132977 12761 hybrid_clock.cc:648] HybridClock initialized: now 1790414769132948 us; error 27 us; skew 500 ppm
I20260926 09:26:09.134361 12761 webserver.cc:460] Webserver started at http://127.24.235.65:38307/ using document root <none> and password file <none>
I20260926 09:26:09.134625 12761 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:09.134686 12761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:09.134891 12761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:09.136011 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data/instance:
uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "b8539bd98c37b4476341c7e0e48a84cc"
server_key_iv: "57bdca90f8c6c7ad06ede70e010d5355"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.136379 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance:
uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "b8539bd98c37b4476341c7e0e48a84cc"
server_key_iv: "57bdca90f8c6c7ad06ede70e010d5355"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.137781 12761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:09.138652 12775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.138876 12761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:09.138949 12761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal
uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "b8539bd98c37b4476341c7e0e48a84cc"
server_key_iv: "57bdca90f8c6c7ad06ede70e010d5355"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.139035 12761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:09.148510 12761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:09.148831 12761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:09.148991 12761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:09.149262 12761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:09.149597 12761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:09.149652 12761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.149693 12761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:09.149726 12761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.156121 12761 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.65:44109
I20260926 09:26:09.156162 12888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.65:44109 every 8 connection(s)
I20260926 09:26:09.156507 12761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/data/info.pb
I20260926 09:26:09.162096 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.157282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.65:40279 (local address 127.24.235.126:33031)
0926 09:26:09.157362 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.157364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.157615 (+ 251us) server_negotiation.cc:407] Connection header received
0926 09:26:09.158195 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.158196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.158220 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.158265 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:09.158953 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.159495 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.160515 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.160789 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.160887 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.161792 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:09.161794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:09.161794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:09.161799 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:09.161814 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:09.161818 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:09.161887 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.161983 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.162013 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 09:26:09.162339 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.157283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33031 (local address 127.24.235.65:40279)
0926 09:26:09.157491 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.157526 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.158075 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.158351 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.158355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.158362 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:09.158857 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.158860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.159601 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.159602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.160405 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.160408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.160904 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.160905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.161086 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.161602 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:09.161602 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:09.161630 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:09.161652 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:09.161887 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:09.161894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:09.161897 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.161987 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.162088 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260926 09:26:09.162827 12889 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33031
I20260926 09:26:09.162943 12889 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:09.163129 12889 heartbeater.cc:511] Master 127.24.235.126:33031 requested a full tablet report, sending...
I20260926 09:26:09.163581 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12761
I20260926 09:26:09.163712 12701 ts_manager.cc:194] Registered new tserver with Master: 5933a5a9ac2947dfa03f8fa6817ab5f5 (127.24.235.65:44109)
I20260926 09:26:09.163697 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-0/wal/instance
I20260926 09:26:09.163951 25517 external_mini_cluster.cc:1428] Setting key 9279b1f3a61d9e6d496bedcacea0aee6
I20260926 09:26:09.164681 12701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.65:40279
I20260926 09:26:09.165329 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.66:0
--local_ip_for_outbound_sockets=127.24.235.66
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--builtin_ntp_servers=127.24.235.84:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 09:26:09.250763 12892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:09.250962 12892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:09.250985 12892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:09.252079 12892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:09.252112 12892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:09.252138 12892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.66
I20260926 09:26:09.253463 12892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.235.66
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.66
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:09.253665 12892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:09.253922 12892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:09.259042 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:09.259029 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:09.259006 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:09.259308 12892 server_base.cc:1034] running on GCE node
I20260926 09:26:09.259521 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:09.259805 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:09.260991 12892 hybrid_clock.cc:648] HybridClock initialized: now 1790414769260990 us; error 48 us; skew 500 ppm
I20260926 09:26:09.262408 12892 webserver.cc:460] Webserver started at http://127.24.235.66:46461/ using document root <none> and password file <none>
I20260926 09:26:09.262663 12892 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:09.262734 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:09.262876 12892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:09.263993 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data/instance:
uuid: "820dbb7023694c2d89f1e6471b6efabd"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "69912c1363745f25bca80eb24accaf7e"
server_key_iv: "763c43ba88ad57f8707b65c3cf41d587"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.264372 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance:
uuid: "820dbb7023694c2d89f1e6471b6efabd"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "69912c1363745f25bca80eb24accaf7e"
server_key_iv: "763c43ba88ad57f8707b65c3cf41d587"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.266000 12892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 09:26:09.266803 12906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.266979 12892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:09.267077 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal
uuid: "820dbb7023694c2d89f1e6471b6efabd"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "69912c1363745f25bca80eb24accaf7e"
server_key_iv: "763c43ba88ad57f8707b65c3cf41d587"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.267182 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:09.281883 12892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:09.282220 12892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:09.282377 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:09.282609 12892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:09.282932 12892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:09.282987 12892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.283025 12892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:09.283052 12892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.289003 12892 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.66:33601
I20260926 09:26:09.289119 13019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.66:33601 every 8 connection(s)
I20260926 09:26:09.289475 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/data/info.pb
I20260926 09:26:09.290660 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 12892
I20260926 09:26:09.290833 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal/instance
I20260926 09:26:09.291018 25517 external_mini_cluster.cc:1428] Setting key 43bb0639495e750f9682249860e68554
I20260926 09:26:09.292172 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.235.67:0
--local_ip_for_outbound_sockets=127.24.235.67
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--builtin_ntp_servers=127.24.235.84:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 09:26:09.295691 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.290308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:55981 (local address 127.24.235.126:33031)
0926 09:26:09.290550 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.290552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.290631 (+ 79us) server_negotiation.cc:407] Connection header received
0926 09:26:09.291279 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.291280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.291311 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.291392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:09.292247 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.293121 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.294146 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.294353 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.294433 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.295386 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:09.295388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:09.295388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:09.295394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:09.295409 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:09.295414 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:09.295468 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.295576 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.295609 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260926 09:26:09.295984 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.290263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33031 (local address 127.24.235.66:55981)
0926 09:26:09.290508 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.290552 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.291155 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.291485 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.291490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.291501 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:09.292130 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.292135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.293226 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.293227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.294043 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.294046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.294452 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.294453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.294596 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.295227 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:09.295227 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:09.295262 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:09.295286 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:09.295477 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:09.295482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:09.295486 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.295578 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.295685 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":124,"threads_started":1}
I20260926 09:26:09.296512 13020 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33031
I20260926 09:26:09.296631 13020 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:09.296875 13020 heartbeater.cc:511] Master 127.24.235.126:33031 requested a full tablet report, sending...
I20260926 09:26:09.297343 12701 ts_manager.cc:194] Registered new tserver with Master: 820dbb7023694c2d89f1e6471b6efabd (127.24.235.66:33601)
I20260926 09:26:09.297856 12701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.66:55981
W20260926 09:26:09.379917 13023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:09.380125 13023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:09.380151 13023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:09.381345 13023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:09.381389 13023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:09.381417 13023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.235.67
I20260926 09:26:09.382833 13023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.235.67
--webserver_port=0
--tserver_master_addrs=127.24.235.126:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.235.67
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:09.383104 13023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:09.383432 13023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:09.388978 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:09.389079 13023 server_base.cc:1034] running on GCE node
W20260926 09:26:09.388995 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:09.388988 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:09.389381 13023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:09.389639 13023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:09.390805 13023 hybrid_clock.cc:648] HybridClock initialized: now 1790414769390791 us; error 26 us; skew 500 ppm
I20260926 09:26:09.392084 13023 webserver.cc:460] Webserver started at http://127.24.235.67:38603/ using document root <none> and password file <none>
I20260926 09:26:09.392351 13023 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:09.392424 13023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:09.392573 13023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:09.393738 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data/instance:
uuid: "75a0f5d18b3c4d64b88c4e7167633565"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "77c979124db316de623b5d6d41f1d5bb"
server_key_iv: "aa45041584dcf71a3616b81094774d94"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.394150 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance:
uuid: "75a0f5d18b3c4d64b88c4e7167633565"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "77c979124db316de623b5d6d41f1d5bb"
server_key_iv: "aa45041584dcf71a3616b81094774d94"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.395470 13023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:09.396193 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.396363 13023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:09.396461 13023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal
uuid: "75a0f5d18b3c4d64b88c4e7167633565"
format_stamp: "Formatted at 2026-09-26 09:26:09 on dist-test-slave-7zlw"
server_key: "77c979124db316de623b5d6d41f1d5bb"
server_key_iv: "aa45041584dcf71a3616b81094774d94"
server_key_version: "encryptionkey@0"
I20260926 09:26:09.396544 13023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:09.416697 13023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:09.417125 13023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:09.417296 13023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:09.417544 13023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 09:26:09.417929 13023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 09:26:09.417984 13023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.418021 13023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 09:26:09.418035 13023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:09.424511 13023 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.67:38087
I20260926 09:26:09.424690 13150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.67:38087 every 8 connection(s)
I20260926 09:26:09.425051 13023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/data/info.pb
I20260926 09:26:09.428300 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13023
I20260926 09:26:09.428428 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal/instance
I20260926 09:26:09.428604 25517 external_mini_cluster.cc:1428] Setting key 5de3533867993cf4481177476bdbff91
I20260926 09:26:09.431128 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.425907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:46465 (local address 127.24.235.126:33031)
0926 09:26:09.425995 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.425997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.426215 (+ 218us) server_negotiation.cc:407] Connection header received
0926 09:26:09.426794 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.426796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.426807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.426856 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:09.427599 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.428138 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.429772 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.429964 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.430020 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.430901 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:09.430902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:09.430902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:09.430906 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:09.430920 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:09.430924 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:09.430969 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.431040 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.431068 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 09:26:09.431411 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.425797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:33031 (local address 127.24.235.67:46465)
0926 09:26:09.426101 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.426139 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.426690 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.426921 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.426924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.426932 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:09.427516 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.427521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.428240 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.428241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.429700 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.429703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.430067 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.430068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.430199 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.430764 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 09:26:09.430764 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:09.430793 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 09:26:09.430818 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:09.430983 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:09.430988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:09.430991 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.431044 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.431127 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260926 09:26:09.431955 13151 heartbeater.cc:347] Connected to a master server at 127.24.235.126:33031
I20260926 09:26:09.432075 13151 heartbeater.cc:464] Registering TS with master...
I20260926 09:26:09.432260 13151 heartbeater.cc:511] Master 127.24.235.126:33031 requested a full tablet report, sending...
I20260926 09:26:09.432649 12701 ts_manager.cc:194] Registered new tserver with Master: 75a0f5d18b3c4d64b88c4e7167633565 (127.24.235.67:38087)
I20260926 09:26:09.433081 12701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.235.67:46465
I20260926 09:26:09.440686 25517 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 09:26:09.445166 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.442224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.24.235.126:33031)
0926 09:26:09.442268 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.442269 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.442375 (+ 106us) server_negotiation.cc:407] Connection header received
0926 09:26:09.442427 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.442428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.442439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.442489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:09.442689 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.443229 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.444016 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.444196 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.444268 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.444902 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:09.444903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:09.444904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:09.444908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:09.444922 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:09.444926 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:09.444968 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.445081 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.445106 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:09.446955 12701 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 09:26:09.448580 12701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 09:26:09.455765 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:38087 (local address 127.0.0.1:40608)
0926 09:26:09.451522 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.451532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.451583 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.452050 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.452051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.452054 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.452110 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.452113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453469 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.453470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.454637 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.454640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.455487 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.455488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.455557 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.455573 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.455647 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.455694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":112,"thread_start_us":107,"threads_started":1}
I20260926 09:26:09.455803 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.24.235.67:38087)
0926 09:26:09.451800 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.451805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.451817 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:26:09.451863 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.451865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.451882 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.451970 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.452193 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453380 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.454739 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.455404 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.455479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.455567 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.455661 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.455711 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260926 09:26:09.456552 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b (DEFAULT_TABLE table=test-table [id=150a3a6951694852b2e3edca91162588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:09.456977 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191ab14e66f34db7b1e31d9e19a93b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:09.457150 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44109 (local address 127.0.0.1:37302)
0926 09:26:09.451391 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.451439 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.451521 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.451809 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.451812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.451815 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.451897 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.451899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453472 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.453473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.454962 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.454964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.456860 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.456862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.456931 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.456935 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.456968 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.457044 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":198,"thread_start_us":81,"threads_started":1}
I20260926 09:26:09.457389 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.24.235.66:33601)
0926 09:26:09.452077 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.452082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.452154 (+ 72us) server_negotiation.cc:407] Connection header received
0926 09:26:09.452273 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.452275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.452288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.452380 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.452511 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453369 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.455908 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.456626 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.456705 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.456794 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.457258 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.457307 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260926 09:26:09.457419 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.24.235.65:44109)
0926 09:26:09.451495 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.451501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.451519 (+ 18us) server_negotiation.cc:407] Connection header received
0926 09:26:09.451655 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.451658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.451675 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.452108 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.452179 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453369 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.455058 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.455812 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.457125 (+ 1313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.457247 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.457274 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.457314 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":123,"threads_started":1}
I20260926 09:26:09.457074 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.451437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33601 (local address 127.0.0.1:56790)
0926 09:26:09.452092 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.452104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.452162 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.452377 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.452378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.452380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.452430 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.452432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.453462 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.453464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.454960 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.454964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.456729 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.456730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.456831 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.456837 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.456899 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.456949 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260926 09:26:09.458261 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b (DEFAULT_TABLE table=test-table [id=150a3a6951694852b2e3edca91162588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:09.458602 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191ab14e66f34db7b1e31d9e19a93b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:09.460021 13165 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Bootstrap starting.
I20260926 09:26:09.460793 12823 tablet_service.cc:1467] Processing CreateTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b (DEFAULT_TABLE table=test-table [id=150a3a6951694852b2e3edca91162588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 09:26:09.460973 13165 tablet_bootstrap.cc:654] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:09.461238 12823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191ab14e66f34db7b1e31d9e19a93b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:09.461411 13165 log.cc:824] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:09.462170 13165 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: No bootstrap required, opened a new log
I20260926 09:26:09.462227 13165 ts_tablet_manager.cc:1397] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 09:26:09.464124 13168 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: Bootstrap starting.
I20260926 09:26:09.464403 13165 raft_consensus.cc:348] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.464560 13165 raft_consensus.cc:374] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:09.464604 13165 raft_consensus.cc:729] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a0f5d18b3c4d64b88c4e7167633565, State: Initialized, Role: FOLLOWER
I20260926 09:26:09.464733 13165 consensus_queue.cc:260] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.464917 13168 tablet_bootstrap.cc:654] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:09.465047 13165 ts_tablet_manager.cc:1428] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 09:26:09.465101 13151 heartbeater.cc:503] Master 127.24.235.126:33031 was elected leader, sending a full tablet report...
I20260926 09:26:09.465090 13167 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Bootstrap starting.
I20260926 09:26:09.465284 13168 log.cc:824] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:09.465914 13167 tablet_bootstrap.cc:654] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:09.466118 13168 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: No bootstrap required, opened a new log
I20260926 09:26:09.466192 13168 ts_tablet_manager.cc:1397] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 09:26:09.466243 13167 log.cc:824] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:09.467142 13167 tablet_bootstrap.cc:492] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: No bootstrap required, opened a new log
I20260926 09:26:09.467235 13167 ts_tablet_manager.cc:1397] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 09:26:09.468319 13168 raft_consensus.cc:348] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.468444 13168 raft_consensus.cc:374] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:09.468466 13168 raft_consensus.cc:729] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5933a5a9ac2947dfa03f8fa6817ab5f5, State: Initialized, Role: FOLLOWER
I20260926 09:26:09.468585 13168 consensus_queue.cc:260] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.468860 13168 ts_tablet_manager.cc:1428] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:09.468915 12889 heartbeater.cc:503] Master 127.24.235.126:33031 was elected leader, sending a full tablet report...
I20260926 09:26:09.469733 13167 raft_consensus.cc:348] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.469923 13167 raft_consensus.cc:374] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:09.469985 13167 raft_consensus.cc:729] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820dbb7023694c2d89f1e6471b6efabd, State: Initialized, Role: FOLLOWER
I20260926 09:26:09.470113 13167 consensus_queue.cc:260] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.470319 13167 ts_tablet_manager.cc:1428] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 09:26:09.470424 13020 heartbeater.cc:503] Master 127.24.235.126:33031 was elected leader, sending a full tablet report...
W20260926 09:26:09.540083 13021 tablet.cc:2369] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 09:26:09.554805 13169 raft_consensus.cc:482] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:09.554987 13169 raft_consensus.cc:504] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.555383 13169 leader_election.cc:288] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5933a5a9ac2947dfa03f8fa6817ab5f5 (127.24.235.65:44109), 820dbb7023694c2d89f1e6471b6efabd (127.24.235.66:33601)
I20260926 09:26:09.556795 13173 raft_consensus.cc:482] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 09:26:09.556926 13173 raft_consensus.cc:504] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.557757 13173 leader_election.cc:288] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5933a5a9ac2947dfa03f8fa6817ab5f5 (127.24.235.65:44109), 75a0f5d18b3c4d64b88c4e7167633565 (127.24.235.67:38087)
I20260926 09:26:09.559473 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.555583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44109 (local address 127.24.235.67:44575)
0926 09:26:09.555684 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.555696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.555754 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.555987 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.555988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.555990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.556100 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.556102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.557155 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.557156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.558454 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.558456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559224 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.559225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.559296 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.559299 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.559347 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.559390 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":56}
I20260926 09:26:09.559559 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.555711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:55221 (local address 127.24.235.66:33601)
0926 09:26:09.555778 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.555780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.555861 (+ 81us) server_negotiation.cc:407] Connection header received
0926 09:26:09.555895 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.555896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.555908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.555966 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.556228 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.557084 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.558588 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559172 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.559223 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.559232 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.559453 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.559490 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260926 09:26:09.559741 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.555646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.67:44575 (local address 127.24.235.65:44109)
0926 09:26:09.555695 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.555697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.555764 (+ 67us) server_negotiation.cc:407] Connection header received
0926 09:26:09.555804 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.555805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.555851 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.555935 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.556228 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.557041 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.558548 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559103 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.559167 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.559177 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.559634 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.559678 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:26:09.559996 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "75a0f5d18b3c4d64b88c4e7167633565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" is_pre_election: true
I20260926 09:26:09.560169 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.555594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.66:33601 (local address 127.24.235.67:55221)
0926 09:26:09.555779 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.555788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.555839 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.555987 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.555988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.555990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.556105 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.556106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.557169 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.557171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.558498 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.558501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559297 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.559298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.559373 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.559377 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.559980 (+ 603us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.560025 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260926 09:26:09.560179 12843 raft_consensus.cc:2454] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a0f5d18b3c4d64b88c4e7167633565 in term 0.
I20260926 09:26:09.560410 13041 leader_election.cc:302] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5933a5a9ac2947dfa03f8fa6817ab5f5, 75a0f5d18b3c4d64b88c4e7167633565; no voters:
I20260926 09:26:09.560596 13169 raft_consensus.cc:2781] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 09:26:09.560680 13169 raft_consensus.cc:482] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 09:26:09.560716 13169 raft_consensus.cc:3037] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:09.560968 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "75a0f5d18b3c4d64b88c4e7167633565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820dbb7023694c2d89f1e6471b6efabd" is_pre_election: true
I20260926 09:26:09.561110 12974 raft_consensus.cc:2454] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a0f5d18b3c4d64b88c4e7167633565 in term 0.
I20260926 09:26:09.561314 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.557727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:57775 (local address 127.24.235.67:38087)
0926 09:26:09.557886 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.557888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.557897 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:09.557917 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.557918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.557927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.557980 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.558174 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.558829 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.560152 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.560729 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.560798 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.560805 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.561224 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.561266 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 09:26:09.561355 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.557524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.67:38087 (local address 127.24.235.66:57775)
0926 09:26:09.557612 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.557622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.557659 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.558020 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.558021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.558023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.558086 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.558088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.558909 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.558910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.560021 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.560024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.560862 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.560863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.560929 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.560932 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.560995 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.561064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260926 09:26:09.561656 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "820dbb7023694c2d89f1e6471b6efabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a0f5d18b3c4d64b88c4e7167633565" is_pre_election: true
I20260926 09:26:09.561759 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.557753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.235.66:42599 (local address 127.24.235.65:44109)
0926 09:26:09.557912 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.557914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.558030 (+ 116us) server_negotiation.cc:407] Connection header received
0926 09:26:09.558142 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.558143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.558153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.558194 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 09:26:09.558441 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559204 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.560742 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.561345 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.561399 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.561406 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.561623 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.561665 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260926 09:26:09.561733 13169 raft_consensus.cc:504] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.561856 13105 raft_consensus.cc:2379] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 820dbb7023694c2d89f1e6471b6efabd in current term 1: Already voted for candidate 75a0f5d18b3c4d64b88c4e7167633565 in this term.
I20260926 09:26:09.561879 13169 leader_election.cc:288] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [CANDIDATE]: Term 1 election: Requested vote from peers 5933a5a9ac2947dfa03f8fa6817ab5f5 (127.24.235.65:44109), 820dbb7023694c2d89f1e6471b6efabd (127.24.235.66:33601)
I20260926 09:26:09.561966 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.557565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44109 (local address 127.24.235.66:42599)
0926 09:26:09.557975 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.557984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:09.558052 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:09.558262 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:09.558263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:09.558265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 09:26:09.558364 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.558365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.559304 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.559304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.560637 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:09.560641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.561470 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.561471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:09.561539 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.561543 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 09:26:09.561832 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 09:26:09.561877 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":85,"threads_started":1}
I20260926 09:26:09.562031 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "75a0f5d18b3c4d64b88c4e7167633565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5"
I20260926 09:26:09.562104 12843 raft_consensus.cc:3037] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:09.562203 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "75a0f5d18b3c4d64b88c4e7167633565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820dbb7023694c2d89f1e6471b6efabd"
I20260926 09:26:09.562268 12974 raft_consensus.cc:3037] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:09.562268 12842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191ab14e66f34db7b1e31d9e19a93b8b" candidate_uuid: "820dbb7023694c2d89f1e6471b6efabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" is_pre_election: true
I20260926 09:26:09.562597 12910 leader_election.cc:302] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 820dbb7023694c2d89f1e6471b6efabd; no voters: 5933a5a9ac2947dfa03f8fa6817ab5f5, 75a0f5d18b3c4d64b88c4e7167633565
I20260926 09:26:09.563107 12843 raft_consensus.cc:2454] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a0f5d18b3c4d64b88c4e7167633565 in term 1.
I20260926 09:26:09.563128 12974 raft_consensus.cc:2454] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a0f5d18b3c4d64b88c4e7167633565 in term 1.
I20260926 09:26:09.563222 13173 raft_consensus.cc:2726] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 09:26:09.563282 13041 leader_election.cc:302] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5933a5a9ac2947dfa03f8fa6817ab5f5, 75a0f5d18b3c4d64b88c4e7167633565; no voters:
I20260926 09:26:09.563352 13169 raft_consensus.cc:2781] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:09.563483 13169 raft_consensus.cc:686] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 LEADER]: Becoming Leader. State: Replica: 75a0f5d18b3c4d64b88c4e7167633565, State: Running, Role: LEADER
I20260926 09:26:09.563581 13169 consensus_queue.cc:237] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } }
I20260926 09:26:09.564512 12701 catalog_manager.cc:5495] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75a0f5d18b3c4d64b88c4e7167633565 (127.24.235.67). New cstate: current_term: 1 leader_uuid: "75a0f5d18b3c4d64b88c4e7167633565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a0f5d18b3c4d64b88c4e7167633565" member_type: VOTER last_known_addr { host: "127.24.235.67" port: 38087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 } health_report { overall_health: UNKNOWN } } }
I20260926 09:26:09.576225 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.573483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.24.235.67:38087)
0926 09:26:09.573567 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:09.573569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:09.573576 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:26:09.573610 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:09.573611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:09.573628 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:09.573703 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:09.573870 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.574419 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.575281 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:09.575533 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:09.575606 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:09.575820 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:09.575960 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:09.576054 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:09.576077 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 09:26:09.576098 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 09:26:09.578233 12843 raft_consensus.cc:1261] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 1 FOLLOWER]: Refusing update from remote peer 75a0f5d18b3c4d64b88c4e7167633565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:09.578236 12974 raft_consensus.cc:1261] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 1 FOLLOWER]: Refusing update from remote peer 75a0f5d18b3c4d64b88c4e7167633565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:09.578593 13177 consensus_queue.cc:1035] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "820dbb7023694c2d89f1e6471b6efabd" member_type: VOTER last_known_addr { host: "127.24.235.66" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:09.578763 13169 consensus_queue.cc:1035] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5933a5a9ac2947dfa03f8fa6817ab5f5" member_type: VOTER last_known_addr { host: "127.24.235.65" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:09.581580 13180 mvcc.cc:205] Tried to move back new op lower bound from 7333538896189759488 to 7333538896133222400. Current Snapshot: MvccSnapshot[applied={T|T < 7333538896189759488}]
I20260926 09:26:09.584416 12701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:09.584514 12701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 09:26:09.585335 12701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3489ed6e6d0240bdabd3651d9a0b606f: Sending DeleteTablet for 3 replicas of tablet 191ab14e66f34db7b1e31d9e19a93b8b
I20260926 09:26:09.585726 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:09 UTC) from {username='slave'} at 127.0.0.1:40608
I20260926 09:26:09.585748 12823 tablet_service.cc:1514] Processing DeleteTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:09 UTC) from {username='slave'} at 127.0.0.1:37302
I20260926 09:26:09.585996 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12761
I20260926 09:26:09.585992 13191 tablet_replica.cc:330] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5: stopping tablet replica
I20260926 09:26:09.586097 13191 raft_consensus.cc:2229] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:09.586069 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 191ab14e66f34db7b1e31d9e19a93b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 09:26:09 UTC) from {username='slave'} at 127.0.0.1:56790
I20260926 09:26:09.586199 13191 raft_consensus.cc:2258] T 191ab14e66f34db7b1e31d9e19a93b8b P 5933a5a9ac2947dfa03f8fa6817ab5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:09.586459 13192 tablet_replica.cc:330] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: stopping tablet replica
I20260926 09:26:09.586599 13192 raft_consensus.cc:2229] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 09:26:09.586748 13192 raft_consensus.cc:2258] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:09.587054 13190 tablet_replica.cc:330] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: stopping tablet replica
I20260926 09:26:09.587152 13190 raft_consensus.cc:2229] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 LEADER]: Raft consensus shutting down.
I20260926 09:26:09.587289 13190 raft_consensus.cc:2258] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 09:26:09.587509 13192 ts_tablet_manager.cc:1905] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:09.587694 13190 ts_tablet_manager.cc:1905] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 09:26:09.589630 13190 ts_tablet_manager.cc:1918] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:09.589721 13190 log.cc:1196] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-2/wal/wals/191ab14e66f34db7b1e31d9e19a93b8b
I20260926 09:26:09.589893 13192 ts_tablet_manager.cc:1918] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 09:26:09.590039 13192 log.cc:1196] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Deleting WAL directory at /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790414522268193-25517-0/minicluster-data/ts-1/wal/wals/191ab14e66f34db7b1e31d9e19a93b8b
I20260926 09:26:09.590175 13190 ts_tablet_manager.cc:1939] T 191ab14e66f34db7b1e31d9e19a93b8b P 75a0f5d18b3c4d64b88c4e7167633565: Deleting consensus metadata
I20260926 09:26:09.590438 13192 ts_tablet_manager.cc:1939] T 191ab14e66f34db7b1e31d9e19a93b8b P 820dbb7023694c2d89f1e6471b6efabd: Deleting consensus metadata
I20260926 09:26:09.590600 12685 catalog_manager.cc:4841] TS 75a0f5d18b3c4d64b88c4e7167633565 (127.24.235.67:38087): tablet 191ab14e66f34db7b1e31d9e19a93b8b (table test-table [id=150a3a6951694852b2e3edca91162588]) successfully deleted
I20260926 09:26:09.590953 12685 catalog_manager.cc:4841] TS 820dbb7023694c2d89f1e6471b6efabd (127.24.235.66:33601): tablet 191ab14e66f34db7b1e31d9e19a93b8b (table test-table [id=150a3a6951694852b2e3edca91162588]) successfully deleted
W20260926 09:26:09.594033 12688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.235.65:44109 (error 108)
I20260926 09:26:09.594203 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12892
I20260926 09:26:09.596251 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:09.596071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.65:44109 (local address 127.0.0.1:37308)
0926 09:26:09.596154 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:09.596207 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 09:26:09.596392 12688 catalog_manager.cc:4584] TS 5933a5a9ac2947dfa03f8fa6817ab5f5 (127.24.235.65:44109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 191ab14e66f34db7b1e31d9e19a93b8b: Network error: Client connection negotiation failed: client connection to 127.24.235.65:44109: connect: Connection refused (error 111)
I20260926 09:26:09.600234 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13023
I20260926 09:26:09.605777 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 12670
2026-09-26T09:26:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (760 ms)
[----------] 6 tests from EncryptionPolicyTest (8517 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:09 dist-test-slave-7zlw krb5kdc[13196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:09 dist-test-slave-7zlw krb5kdc[13196](info): set up 2 sockets
Sep 26 09:26:09 dist-test-slave-7zlw krb5kdc[13196](info): commencing operation
krb5kdc: starting...
W20260926 09:26:10.653263 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:10 dist-test-slave-7zlw krb5kdc[13196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:10.780558 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:10.865201 13209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:10.865494 13209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:10.865520 13209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:10.866722 13209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:10.866766 13209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:10.866781 13209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:10.866793 13209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:10.866806 13209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:10.868321 13209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:10.868580 13209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:10.868826 13209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:10.874123 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:10.874181 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:10.874193 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:10.874362 13209 server_base.cc:1034] running on GCE node
I20260926 09:26:10.874719 13209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:10.874996 13209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:10.876155 13209 hybrid_clock.cc:648] HybridClock initialized: now 1790414770876139 us; error 33 us; skew 500 ppm
Sep 26 09:26:10 dist-test-slave-7zlw krb5kdc[13196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:10.879144 13209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:10.879626 13209 webserver.cc:460] Webserver started at http://127.0.0.1:38781/ using document root <none> and password file <none>
I20260926 09:26:10.879838 13209 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:10.879890 13209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:10.879995 13209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:10.881287 13209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "52e564aecb144f9cbfc3d8822eb8f387"
format_stamp: "Formatted at 2026-09-26 09:26:10 on dist-test-slave-7zlw"
server_key: "ecc9c1f87c8f3bf90be7b5c7db7dffc9"
server_key_iv: "8443dfcfbc74085071d138ef1f03605d"
server_key_version: "encryptionkey@0"
I20260926 09:26:10.881688 13209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "52e564aecb144f9cbfc3d8822eb8f387"
format_stamp: "Formatted at 2026-09-26 09:26:10 on dist-test-slave-7zlw"
server_key: "ecc9c1f87c8f3bf90be7b5c7db7dffc9"
server_key_iv: "8443dfcfbc74085071d138ef1f03605d"
server_key_version: "encryptionkey@0"
I20260926 09:26:10.883251 13209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:10.884110 13224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:10.884356 13209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:10.884472 13209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "52e564aecb144f9cbfc3d8822eb8f387"
format_stamp: "Formatted at 2026-09-26 09:26:10 on dist-test-slave-7zlw"
server_key: "ecc9c1f87c8f3bf90be7b5c7db7dffc9"
server_key_iv: "8443dfcfbc74085071d138ef1f03605d"
server_key_version: "encryptionkey@0"
I20260926 09:26:10.884558 13209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:10.901288 13209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:10.904083 13209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:10.904255 13209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:10.909439 13209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40237
I20260926 09:26:10.909485 13276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40237 every 8 connection(s)
I20260926 09:26:10.909894 13209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:10.910808 13277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:10.913254 13277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387: Bootstrap starting.
I20260926 09:26:10.913872 13277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:10.914171 13277 log.cc:824] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:10.915030 13277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387: No bootstrap required, opened a new log
I20260926 09:26:10.916296 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13209
I20260926 09:26:10.916452 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:10.916702 25517 external_mini_cluster.cc:1428] Setting key c6e3ebd256a511d321cd9fedf157d5e3
I20260926 09:26:10.916838 13277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260926 09:26:10.917068 13277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:10.917124 13277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e564aecb144f9cbfc3d8822eb8f387, State: Initialized, Role: FOLLOWER
I20260926 09:26:10.917292 13277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260926 09:26:10.917385 13277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:10.917418 13277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:10.917459 13277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:10.918177 13277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260926 09:26:10.918298 13277 leader_election.cc:302] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e564aecb144f9cbfc3d8822eb8f387; no voters:
I20260926 09:26:10.918442 13277 leader_election.cc:288] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:10.918457 13282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:10.918563 13282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [term 1 LEADER]: Becoming Leader. State: Replica: 52e564aecb144f9cbfc3d8822eb8f387, State: Running, Role: LEADER
I20260926 09:26:10.918639 13282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260926 09:26:10.918707 13277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:10.918979 13282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e564aecb144f9cbfc3d8822eb8f387. Latest consensus state: current_term: 1 leader_uuid: "52e564aecb144f9cbfc3d8822eb8f387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260926 09:26:10.919055 13282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:10.918990 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e564aecb144f9cbfc3d8822eb8f387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e564aecb144f9cbfc3d8822eb8f387" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260926 09:26:10.919255 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:10.919271 13286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:10.919916 13286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:10.921854 13286 catalog_manager.cc:1347] Generated new cluster ID: ee2c0ee9e7b549c385771882117cef09
I20260926 09:26:10.921907 13286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:10 dist-test-slave-7zlw krb5kdc[13196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:26:10.929845 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:10.917375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47911 (local address 127.0.0.1:40237)
0926 09:26:10.917626 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:10.917635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:10.917669 (+ 34us) server_negotiation.cc:407] Connection header received
0926 09:26:10.918095 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:10.918105 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:10.918172 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:10.918327 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:10.919253 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:10.920836 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:10.921801 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:10.922383 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:10.922481 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:10.925323 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:10.925327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:10.925328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:10.925339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:10.927874 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:10.928311 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:10.928314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:10.928314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:10.928414 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:10.928650 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:10.928650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:10.928651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:10.928946 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:10.929102 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:10.929509 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 09:26:10.929614 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":195,"thread_start_us":139,"threads_started":1}
I20260926 09:26:10.930850 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:10.939530 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:10.932392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57497 (local address 127.0.0.1:40237)
0926 09:26:10.932428 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:10.932430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:10.932457 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:26:10.932513 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:10.932514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:10.932524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:10.932582 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:10.933182 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:10.933950 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:10.934906 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:10.935245 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:10.935316 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:10.936636 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:10.936638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:10.936638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:10.936645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:10.938244 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:10.938644 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:10.938645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:10.938645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:10.938704 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:10.938923 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:10.938924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:10.938924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:10.939054 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:10.939158 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:10.939278 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 09:26:10.939339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:10.949246 13286 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:10.949966 13286 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:10.956646 13286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e564aecb144f9cbfc3d8822eb8f387: Generated new TSK 0
I20260926 09:26:10.956915 13286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:10.960700 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13209
W20260926 09:26:10.965759 13194 connection.cc:535] client connection to 127.0.0.1:40237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40237): Connection reset by peer (error 104)
2026-09-26T09:26:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:11 dist-test-slave-7zlw krb5kdc[13308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:11 dist-test-slave-7zlw krb5kdc[13308](info): set up 2 sockets
Sep 26 09:26:11 dist-test-slave-7zlw krb5kdc[13308](info): commencing operation
krb5kdc: starting...
W20260926 09:26:12.041908 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:12 dist-test-slave-7zlw krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:12.168197 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:12.253705 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:12.253921 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:12.253945 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:12.255072 13321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:12.255110 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:12.255124 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:12.255136 13321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:12.255149 13321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:12.256618 13321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:12.256820 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:12.256999 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:12.261977 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:12.262120 13321 server_base.cc:1034] running on GCE node
W20260926 09:26:12.261971 13326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:12.262069 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:12.262487 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:12.262786 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:12.264019 13321 hybrid_clock.cc:648] HybridClock initialized: now 1790414772263999 us; error 42 us; skew 500 ppm
Sep 26 09:26:12 dist-test-slave-7zlw krb5kdc[13308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:12.271253 13321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:12.271795 13321 webserver.cc:460] Webserver started at http://127.0.0.1:46713/ using document root <none> and password file <none>
I20260926 09:26:12.272073 13321 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:12.272147 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:12.272306 13321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:12.273506 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "a5cf349ee48347098e99e619fef78e00"
format_stamp: "Formatted at 2026-09-26 09:26:12 on dist-test-slave-7zlw"
server_key: "005d04c47c2ab8ddfbe58e6b2deb03e7"
server_key_iv: "585c402e46642c22cadab57249d7a276"
server_key_version: "encryptionkey@0"
I20260926 09:26:12.273886 13321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "a5cf349ee48347098e99e619fef78e00"
format_stamp: "Formatted at 2026-09-26 09:26:12 on dist-test-slave-7zlw"
server_key: "005d04c47c2ab8ddfbe58e6b2deb03e7"
server_key_iv: "585c402e46642c22cadab57249d7a276"
server_key_version: "encryptionkey@0"
I20260926 09:26:12.275310 13321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:12.276065 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:12.276252 13321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 09:26:12.276347 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "a5cf349ee48347098e99e619fef78e00"
format_stamp: "Formatted at 2026-09-26 09:26:12 on dist-test-slave-7zlw"
server_key: "005d04c47c2ab8ddfbe58e6b2deb03e7"
server_key_iv: "585c402e46642c22cadab57249d7a276"
server_key_version: "encryptionkey@0"
I20260926 09:26:12.276430 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:12.294035 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:12.296226 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:12.296406 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:12.300921 13321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44979
I20260926 09:26:12.301090 13388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44979 every 8 connection(s)
I20260926 09:26:12.301405 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:12.302341 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:12.303843 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13321
I20260926 09:26:12.303967 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:12.304175 25517 external_mini_cluster.cc:1428] Setting key 2a772eee560092f7d1cfa44107c129cd
I20260926 09:26:12.305279 13389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00: Bootstrap starting.
I20260926 09:26:12.305989 13389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:12.306298 13389 log.cc:824] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:12.307016 13389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00: No bootstrap required, opened a new log
I20260926 09:26:12.309420 13389 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260926 09:26:12.309631 13389 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:12.309736 13389 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cf349ee48347098e99e619fef78e00, State: Initialized, Role: FOLLOWER
I20260926 09:26:12.309908 13389 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260926 09:26:12.309958 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:12.304729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42191 (local address 127.0.0.1:44979)
0926 09:26:12.304913 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:12.304921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:12.304949 (+ 28us) server_negotiation.cc:407] Connection header received
0926 09:26:12.305438 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:12.305446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:12.305495 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:12.305611 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:12.305864 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:12.306853 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:12.307917 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:12.308346 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:12.308430 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:12.309223 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:12.309227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:12.309227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:12.309389 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:12.309423 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:12.309433 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:12.309501 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:12.309652 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 09:26:12.309694 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260926 09:26:12.309990 13389 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:12.310015 13389 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:12.310073 13389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:12.310807 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:12.310814 13389 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260926 09:26:12.310935 13389 leader_election.cc:302] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5cf349ee48347098e99e619fef78e00; no voters:
I20260926 09:26:12.311141 13389 leader_election.cc:288] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:12.311199 13394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:12.311326 13394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [term 1 LEADER]: Becoming Leader. State: Replica: a5cf349ee48347098e99e619fef78e00, State: Running, Role: LEADER
I20260926 09:26:12.311393 13389 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:12.311462 13394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } }
I20260926 09:26:12.312086 13396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5cf349ee48347098e99e619fef78e00. Latest consensus state: current_term: 1 leader_uuid: "a5cf349ee48347098e99e619fef78e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } } }
I20260926 09:26:12.312186 13396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:12.312357 13395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5cf349ee48347098e99e619fef78e00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cf349ee48347098e99e619fef78e00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44979 } } }
I20260926 09:26:12.312399 13407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:12.312417 13395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:12.312903 13407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:12.314553 13407 catalog_manager.cc:1347] Generated new cluster ID: 54891e83e88b45b1978bd4064af76b03
I20260926 09:26:12.314610 13407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:12.316417 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:12.312328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44769 (local address 127.0.0.1:44979)
0926 09:26:12.313106 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:12.313108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:12.313117 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:12.313138 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:12.313139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:12.313151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:12.313212 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:12.313384 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:12.314094 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:12.315156 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:12.315369 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:12.315426 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:12.316169 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:12.316170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:12.316171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:12.316176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:12.316194 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:12.316198 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:12.316265 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:12.316337 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 09:26:12.316349 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761}
I20260926 09:26:12.323392 13407 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:12.323935 13407 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:12.329488 13407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5cf349ee48347098e99e619fef78e00: Generated new TSK 0
I20260926 09:26:12.329708 13407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:12.333410 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13321
W20260926 09:26:12.338421 13306 connection.cc:535] client connection to 127.0.0.1:44979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44979): Connection reset by peer (error 104)
2026-09-26T09:26:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:12 dist-test-slave-7zlw krb5kdc[13420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:12 dist-test-slave-7zlw krb5kdc[13420](info): set up 2 sockets
Sep 26 09:26:12 dist-test-slave-7zlw krb5kdc[13420](info): commencing operation
krb5kdc: starting...
W20260926 09:26:13.381392 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:13 dist-test-slave-7zlw krb5kdc[13420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:13.508481 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:13.592533 13433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:13.592795 13433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:13.592834 13433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:13.594080 13433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:13.594158 13433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:13.594183 13433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:13.594218 13433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:13.594244 13433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:13.595744 13433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:13.596022 13433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:13.596266 13433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:13.601397 13433 server_base.cc:1034] running on GCE node
W20260926 09:26:13.601366 13441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:13.601336 13438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:13.601351 13439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:13.601858 13433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:13.602164 13433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:13.603327 13433 hybrid_clock.cc:648] HybridClock initialized: now 1790414773603306 us; error 40 us; skew 500 ppm
Sep 26 09:26:13 dist-test-slave-7zlw krb5kdc[13420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:13.605903 13433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:13.606384 13433 webserver.cc:460] Webserver started at http://127.0.0.1:40481/ using document root <none> and password file <none>
I20260926 09:26:13.606632 13433 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:13.606701 13433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:13.606850 13433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:13.607908 13433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "6c84ee0443514a77b5f12e5a121d3d10"
format_stamp: "Formatted at 2026-09-26 09:26:13 on dist-test-slave-7zlw"
server_key: "e84560664b9e71ccf00a2de18a86697e"
server_key_iv: "d87c05823bd49e7792fb8a1f487427be"
server_key_version: "encryptionkey@0"
I20260926 09:26:13.608291 13433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "6c84ee0443514a77b5f12e5a121d3d10"
format_stamp: "Formatted at 2026-09-26 09:26:13 on dist-test-slave-7zlw"
server_key: "e84560664b9e71ccf00a2de18a86697e"
server_key_iv: "d87c05823bd49e7792fb8a1f487427be"
server_key_version: "encryptionkey@0"
I20260926 09:26:13.609710 13433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:13.610576 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:13.610839 13433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:13.610944 13433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "6c84ee0443514a77b5f12e5a121d3d10"
format_stamp: "Formatted at 2026-09-26 09:26:13 on dist-test-slave-7zlw"
server_key: "e84560664b9e71ccf00a2de18a86697e"
server_key_iv: "d87c05823bd49e7792fb8a1f487427be"
server_key_version: "encryptionkey@0"
I20260926 09:26:13.611027 13433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:13.614503 13433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:13.616828 13433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:13.616986 13433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:13.621479 13433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38263
I20260926 09:26:13.621515 13500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38263 every 8 connection(s)
I20260926 09:26:13.621842 13433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:13.622857 13501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:13.623600 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13433
I20260926 09:26:13.623724 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:13.623904 25517 external_mini_cluster.cc:1428] Setting key c26f4a4c61b45be6da2007cba0ac4354
I20260926 09:26:13.625964 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10: Bootstrap starting.
I20260926 09:26:13.626466 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:13.624416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41069 (local address 127.0.0.1:38263)
0926 09:26:13.624620 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:13.624631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:13.624667 (+ 36us) server_negotiation.cc:407] Connection header received
0926 09:26:13.625135 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:13.625141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:13.625167 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:13.625319 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:13.625990 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:13.625994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:13.625994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:13.625999 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:13.626048 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:13.626098 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:13.626193 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:13.626261 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 09:26:13.626302 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260926 09:26:13.626729 13501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:13.627058 13501 log.cc:824] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:13.627454 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:13.627833 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10: No bootstrap required, opened a new log
I20260926 09:26:13.630342 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:13.629108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35573 (local address 127.0.0.1:38263)
0926 09:26:13.629150 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:13.629152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:13.629418 (+ 266us) server_negotiation.cc:407] Connection header received
0926 09:26:13.629475 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:13.629476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:13.629480 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:13.629526 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:13.630060 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:13.630061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:13.630061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:13.630065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:13.630077 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:13.630081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:13.630140 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:13.630243 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 09:26:13.630253 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:13.630709 13501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } }
I20260926 09:26:13.630867 13501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:13.630911 13501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c84ee0443514a77b5f12e5a121d3d10, State: Initialized, Role: FOLLOWER
I20260926 09:26:13.631029 13501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } }
I20260926 09:26:13.631134 13501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:13.631177 13501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:13.631227 13501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:13.631765 13501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } }
I20260926 09:26:13.631884 13501 leader_election.cc:302] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c84ee0443514a77b5f12e5a121d3d10; no voters:
I20260926 09:26:13.632035 13501 leader_election.cc:288] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:13.632081 13511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:13.632277 13511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [term 1 LEADER]: Becoming Leader. State: Replica: 6c84ee0443514a77b5f12e5a121d3d10, State: Running, Role: LEADER
I20260926 09:26:13.632313 13501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:13.632429 13511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } }
I20260926 09:26:13.632812 13513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c84ee0443514a77b5f12e5a121d3d10. Latest consensus state: current_term: 1 leader_uuid: "6c84ee0443514a77b5f12e5a121d3d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } } }
I20260926 09:26:13.632938 13513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:13.632805 13512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c84ee0443514a77b5f12e5a121d3d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c84ee0443514a77b5f12e5a121d3d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38263 } } }
I20260926 09:26:13.633107 13512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:13.633280 13518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:13.634021 13518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:13.635636 13518 catalog_manager.cc:1347] Generated new cluster ID: 43b047cd8faa4c93a4d3c774a0cd4a35
I20260926 09:26:13.635699 13518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:13.665098 13518 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:13.665293 13518 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:13.667949 13518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c84ee0443514a77b5f12e5a121d3d10: Generated new TSK 0
I20260926 09:26:13.668129 13518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:13.673950 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13433
2026-09-26T09:26:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:13 dist-test-slave-7zlw krb5kdc[13532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:13 dist-test-slave-7zlw krb5kdc[13532](info): set up 2 sockets
Sep 26 09:26:13 dist-test-slave-7zlw krb5kdc[13532](info): commencing operation
krb5kdc: starting...
W20260926 09:26:14.721539 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:14 dist-test-slave-7zlw krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:14.848948 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:14.932787 13545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:14.933087 13545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:14.933137 13545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:14.934412 13545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:14.934492 13545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:14.934518 13545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:14.934563 13545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:14.934584 13545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:14.936131 13545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:14.936445 13545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:14.936688 13545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:14.941970 13553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:14.942014 13551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:14.942003 13550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:14.942090 13545 server_base.cc:1034] running on GCE node
I20260926 09:26:14.942480 13545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:14.942864 13545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:14.944036 13545 hybrid_clock.cc:648] HybridClock initialized: now 1790414774944007 us; error 36 us; skew 500 ppm
Sep 26 09:26:14 dist-test-slave-7zlw krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:14.947047 13545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:14.947585 13545 webserver.cc:460] Webserver started at http://127.0.0.1:45891/ using document root <none> and password file <none>
I20260926 09:26:14.947870 13545 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:14.947950 13545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:14.948133 13545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:14.949504 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "a1b974d16ca24c57851e56258f770a72"
format_stamp: "Formatted at 2026-09-26 09:26:14 on dist-test-slave-7zlw"
server_key: "ffff204ee207f66c1034b3cb5b11a524"
server_key_iv: "c0d331a8bc18eb45f520a2d1fcbce446"
server_key_version: "encryptionkey@0"
I20260926 09:26:14.949945 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "a1b974d16ca24c57851e56258f770a72"
format_stamp: "Formatted at 2026-09-26 09:26:14 on dist-test-slave-7zlw"
server_key: "ffff204ee207f66c1034b3cb5b11a524"
server_key_iv: "c0d331a8bc18eb45f520a2d1fcbce446"
server_key_version: "encryptionkey@0"
I20260926 09:26:14.951583 13545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:14.952498 13560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:14.952718 13545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:14.952831 13545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "a1b974d16ca24c57851e56258f770a72"
format_stamp: "Formatted at 2026-09-26 09:26:14 on dist-test-slave-7zlw"
server_key: "ffff204ee207f66c1034b3cb5b11a524"
server_key_iv: "c0d331a8bc18eb45f520a2d1fcbce446"
server_key_version: "encryptionkey@0"
I20260926 09:26:14.952912 13545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:14.979238 13545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:14.981374 13545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:14.981544 13545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:14.985968 13545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35043
I20260926 09:26:14.986018 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35043 every 8 connection(s)
I20260926 09:26:14.986418 13545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:14.987407 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:14.989848 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72: Bootstrap starting.
I20260926 09:26:14.990420 13613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:14.990659 13613 log.cc:824] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:14.991405 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72: No bootstrap required, opened a new log
I20260926 09:26:14.993040 13613 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } }
I20260926 09:26:14.993238 13613 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:14.993273 13613 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b974d16ca24c57851e56258f770a72, State: Initialized, Role: FOLLOWER
I20260926 09:26:14.993430 13613 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } }
I20260926 09:26:14.993510 13613 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:14.993532 13613 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:14.993561 13613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:14.994087 13613 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } }
I20260926 09:26:14.994171 13613 leader_election.cc:302] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b974d16ca24c57851e56258f770a72; no voters:
I20260926 09:26:14.994297 13613 leader_election.cc:288] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:14.994382 13616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:14.994578 13616 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [term 1 LEADER]: Becoming Leader. State: Replica: a1b974d16ca24c57851e56258f770a72, State: Running, Role: LEADER
I20260926 09:26:14.994624 13613 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:14.994746 13616 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } }
I20260926 09:26:14.994803 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13545
I20260926 09:26:14.994910 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:14.995116 25517 external_mini_cluster.cc:1428] Setting key d5d50a64c82ddc463a1e99e1713b8f0e
I20260926 09:26:14.995406 13618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b974d16ca24c57851e56258f770a72. Latest consensus state: current_term: 1 leader_uuid: "a1b974d16ca24c57851e56258f770a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } } }
I20260926 09:26:14.995501 13618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:14.995702 13617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b974d16ca24c57851e56258f770a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b974d16ca24c57851e56258f770a72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35043 } } }
I20260926 09:26:14.995790 13617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:14.996198 13622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:14.996744 13622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:14.999770 13622 catalog_manager.cc:1347] Generated new cluster ID: 8099c12e1e004fbb974138a7eed056f2
I20260926 09:26:14.999826 13622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:15 dist-test-slave-7zlw krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:26:15.010447 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:14.995784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42973 (local address 127.0.0.1:35043)
0926 09:26:14.996102 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:14.996111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:14.996271 (+ 160us) server_negotiation.cc:407] Connection header received
0926 09:26:14.996523 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:14.996530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:14.996589 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:14.996745 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:14.997850 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:14.999613 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:15.000987 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:15.001890 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:15.002013 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:15.006417 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:15.006421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:15.006422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:15.006434 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:15.008540 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:15.009046 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:15.009049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:15.009050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:15.009171 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:15.009423 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:15.009424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:15.009424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:15.009743 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:15.009904 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:15.010052 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:26:15.010193 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":64,"threads_started":1}
I20260926 09:26:15.011401 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:15.016597 13622 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:15.017410 13622 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:15.020551 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:15.012875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48027 (local address 127.0.0.1:35043)
0926 09:26:15.012914 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:15.012916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:15.013152 (+ 236us) server_negotiation.cc:407] Connection header received
0926 09:26:15.013217 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:15.013218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:15.013227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:15.013274 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:15.014006 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:15.014751 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:15.015923 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:15.016230 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:15.016322 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:15.017871 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:15.017872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:15.017872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:15.017879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:15.019433 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:15.019793 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:15.019794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:15.019795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:15.019842 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:15.020005 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:15.020006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:15.020006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:15.020112 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:15.020188 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:15.020372 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 09:26:15.020462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:15.024557 13622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b974d16ca24c57851e56258f770a72: Generated new TSK 0
I20260926 09:26:15.024755 13622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:15.027496 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13545
2026-09-26T09:26:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:15 dist-test-slave-7zlw krb5kdc[13644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:15 dist-test-slave-7zlw krb5kdc[13644](info): set up 2 sockets
Sep 26 09:26:15 dist-test-slave-7zlw krb5kdc[13644](info): commencing operation
krb5kdc: starting...
W20260926 09:26:16.061362 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:16 dist-test-slave-7zlw krb5kdc[13644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:16.188143 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:16.272158 13657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:16.272433 13657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:16.272475 13657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:16.273790 13657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:16.273864 13657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:16.273888 13657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:16.273911 13657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:16.273941 13657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:16.275511 13657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:16.275795 13657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:16.276049 13657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:16.281203 13665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:16.281203 13663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:16.281240 13662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:16.281342 13657 server_base.cc:1034] running on GCE node
I20260926 09:26:16.281733 13657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:16.282037 13657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:16.283219 13657 hybrid_clock.cc:648] HybridClock initialized: now 1790414776283193 us; error 30 us; skew 500 ppm
Sep 26 09:26:16 dist-test-slave-7zlw krb5kdc[13644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:16.286015 13657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:16.286471 13657 webserver.cc:460] Webserver started at http://127.0.0.1:46703/ using document root <none> and password file <none>
I20260926 09:26:16.286761 13657 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:16.286837 13657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:16.286988 13657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:16.288102 13657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "d352914b9bd84c138dba3987c5b1dd8e"
format_stamp: "Formatted at 2026-09-26 09:26:16 on dist-test-slave-7zlw"
server_key: "8b4652c635472fa271a3ccb6831572a8"
server_key_iv: "14b906824d90bcbad5b48097f7cc0a8f"
server_key_version: "encryptionkey@0"
I20260926 09:26:16.288488 13657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "d352914b9bd84c138dba3987c5b1dd8e"
format_stamp: "Formatted at 2026-09-26 09:26:16 on dist-test-slave-7zlw"
server_key: "8b4652c635472fa271a3ccb6831572a8"
server_key_iv: "14b906824d90bcbad5b48097f7cc0a8f"
server_key_version: "encryptionkey@0"
I20260926 09:26:16.290001 13657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:16.290963 13672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:16.291180 13657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:16.291311 13657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "d352914b9bd84c138dba3987c5b1dd8e"
format_stamp: "Formatted at 2026-09-26 09:26:16 on dist-test-slave-7zlw"
server_key: "8b4652c635472fa271a3ccb6831572a8"
server_key_iv: "14b906824d90bcbad5b48097f7cc0a8f"
server_key_version: "encryptionkey@0"
I20260926 09:26:16.291393 13657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:16.320065 13657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:16.322563 13657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:16.322745 13657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:16.327088 13657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43317
I20260926 09:26:16.327142 13724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43317 every 8 connection(s)
I20260926 09:26:16.327538 13657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:16.328522 13725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:16.330999 13725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e: Bootstrap starting.
I20260926 09:26:16.331634 13725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:16.331947 13725 log.cc:824] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:16.332706 13725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e: No bootstrap required, opened a new log
I20260926 09:26:16.333598 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13657
I20260926 09:26:16.333721 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:16.333909 25517 external_mini_cluster.cc:1428] Setting key a16c78ec1f6d05885b89e69ca93f5882
I20260926 09:26:16.335157 13725 raft_consensus.cc:348] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260926 09:26:16.335340 13725 raft_consensus.cc:374] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:16.335379 13725 raft_consensus.cc:729] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d352914b9bd84c138dba3987c5b1dd8e, State: Initialized, Role: FOLLOWER
I20260926 09:26:16.335539 13725 consensus_queue.cc:260] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260926 09:26:16.335608 13725 raft_consensus.cc:388] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:16.335629 13725 raft_consensus.cc:482] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:16.335706 13725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:16.336508 13725 raft_consensus.cc:504] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260926 09:26:16.336621 13725 leader_election.cc:302] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d352914b9bd84c138dba3987c5b1dd8e; no voters:
I20260926 09:26:16.336776 13725 leader_election.cc:288] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:16.336862 13730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:16.337055 13730 raft_consensus.cc:686] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [term 1 LEADER]: Becoming Leader. State: Replica: d352914b9bd84c138dba3987c5b1dd8e, State: Running, Role: LEADER
I20260926 09:26:16.337154 13725 sys_catalog.cc:564] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:16.337176 13730 consensus_queue.cc:237] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260926 09:26:16.337698 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d352914b9bd84c138dba3987c5b1dd8e. Latest consensus state: current_term: 1 leader_uuid: "d352914b9bd84c138dba3987c5b1dd8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260926 09:26:16.337822 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:16.338270 13732 sys_catalog.cc:455] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d352914b9bd84c138dba3987c5b1dd8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d352914b9bd84c138dba3987c5b1dd8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260926 09:26:16.338403 13732 sys_catalog.cc:458] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:16.338536 13739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:16.339465 13739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:16.341216 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:16.334482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50855 (local address 127.0.0.1:43317)
0926 09:26:16.334727 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:16.334737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:16.334777 (+ 40us) server_negotiation.cc:407] Connection header received
0926 09:26:16.335110 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:16.335119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:16.335182 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:16.335344 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:16.335702 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:16.337436 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:16.338648 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:16.339251 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:16.339361 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:16.340253 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:16.340257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:16.340258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:16.340483 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:16.340523 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:16.340539 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:16.340611 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:16.340913 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 09:26:16.340977 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260926 09:26:16.341406 13739 catalog_manager.cc:1347] Generated new cluster ID: 0e7ffd8f4b9c4aacb10bd32725180b9d
I20260926 09:26:16.341483 13739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:16.342183 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:16.347975 13739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:16.348207 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:16.343579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54491 (local address 127.0.0.1:43317)
0926 09:26:16.343619 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:16.343621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:16.345224 (+ 1603us) server_negotiation.cc:407] Connection header received
0926 09:26:16.345297 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:16.345298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:16.345310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:16.345355 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:16.345509 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:16.345989 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:16.346819 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:16.347120 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:16.347199 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:16.347920 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:16.347921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:16.347922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:16.347927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:16.347941 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:16.347946 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:16.348015 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:16.348077 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 09:26:16.348102 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260926 09:26:16.348806 13739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:16.361478 13739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d352914b9bd84c138dba3987c5b1dd8e: Generated new TSK 0
I20260926 09:26:16.361737 13739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:16.364485 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13657
2026-09-26T09:26:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:16 dist-test-slave-7zlw krb5kdc[13756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:16 dist-test-slave-7zlw krb5kdc[13756](info): set up 2 sockets
Sep 26 09:26:16 dist-test-slave-7zlw krb5kdc[13756](info): commencing operation
krb5kdc: starting...
W20260926 09:26:17.425367 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:17 dist-test-slave-7zlw krb5kdc[13756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:17.552780 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:17.634583 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:17.634835 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:17.634876 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:17.636029 13769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:17.636098 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:17.636122 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:17.636150 13769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:17.636181 13769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:17.637686 13769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:17.637965 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:17.638198 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:17.643211 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:17.643281 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:17.643501 13769 server_base.cc:1034] running on GCE node
W20260926 09:26:17.643555 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:17.643838 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:17.644166 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:17.645341 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790414777645328 us; error 46 us; skew 500 ppm
Sep 26 09:26:17 dist-test-slave-7zlw krb5kdc[13756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:17.652648 13769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:17.653206 13769 webserver.cc:460] Webserver started at http://127.0.0.1:45089/ using document root <none> and password file <none>
I20260926 09:26:17.653473 13769 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:17.653544 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:17.653692 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:17.654850 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "f6a4f4996bd747dbac40f16cc1244268"
format_stamp: "Formatted at 2026-09-26 09:26:17 on dist-test-slave-7zlw"
server_key: "9ebcf65191ed712718a7469e981dfaca"
server_key_iv: "7f03e99464688cf02aeb75df1064261f"
server_key_version: "encryptionkey@0"
I20260926 09:26:17.655221 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "f6a4f4996bd747dbac40f16cc1244268"
format_stamp: "Formatted at 2026-09-26 09:26:17 on dist-test-slave-7zlw"
server_key: "9ebcf65191ed712718a7469e981dfaca"
server_key_iv: "7f03e99464688cf02aeb75df1064261f"
server_key_version: "encryptionkey@0"
I20260926 09:26:17.656632 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:17.657475 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:17.657655 13769 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:17.657753 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "f6a4f4996bd747dbac40f16cc1244268"
format_stamp: "Formatted at 2026-09-26 09:26:17 on dist-test-slave-7zlw"
server_key: "9ebcf65191ed712718a7469e981dfaca"
server_key_iv: "7f03e99464688cf02aeb75df1064261f"
server_key_version: "encryptionkey@0"
I20260926 09:26:17.657840 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:17.661341 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:17.663232 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:17.663416 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:17.667446 13769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33447
I20260926 09:26:17.667477 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33447 every 8 connection(s)
I20260926 09:26:17.667806 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:17.668964 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:17.671311 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268: Bootstrap starting.
I20260926 09:26:17.671936 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:17.672236 13837 log.cc:824] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:17.672961 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268: No bootstrap required, opened a new log
I20260926 09:26:17.675112 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 09:26:17.675253 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:17.675275 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a4f4996bd747dbac40f16cc1244268, State: Initialized, Role: FOLLOWER
I20260926 09:26:17.675423 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 09:26:17.675524 13837 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:17.675547 13837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:17.675581 13837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:17.676079 13837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 09:26:17.676162 13837 leader_election.cc:302] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a4f4996bd747dbac40f16cc1244268; no voters:
I20260926 09:26:17.676281 13837 leader_election.cc:288] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:17.676350 13840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:17.676527 13840 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [term 1 LEADER]: Becoming Leader. State: Replica: f6a4f4996bd747dbac40f16cc1244268, State: Running, Role: LEADER
I20260926 09:26:17.676560 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:17.676614 13840 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } }
I20260926 09:26:17.677313 13842 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a4f4996bd747dbac40f16cc1244268. Latest consensus state: current_term: 1 leader_uuid: "f6a4f4996bd747dbac40f16cc1244268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } } }
I20260926 09:26:17.677414 13842 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:17.677644 13849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:17.678002 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13769
I20260926 09:26:17.678145 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:17.678329 25517 external_mini_cluster.cc:1428] Setting key b496dc7bbbc75b0d328d6cb4b237d0e0
I20260926 09:26:17.677817 13841 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a4f4996bd747dbac40f16cc1244268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a4f4996bd747dbac40f16cc1244268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33447 } } }
I20260926 09:26:17.678905 13841 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:17.679400 13849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:17.681113 13849 catalog_manager.cc:1347] Generated new cluster ID: 25c1fe7931f34f49b4a166d957be26d1
I20260926 09:26:17.681165 13849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:17.681169 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:17.678847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53451 (local address 127.0.0.1:33447)
0926 09:26:17.679256 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:17.679267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:17.679305 (+ 38us) server_negotiation.cc:407] Connection header received
0926 09:26:17.679423 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:17.679430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:17.679464 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:17.679876 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:17.680712 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:17.680715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:17.680716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:17.680719 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:17.680754 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:17.680764 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:17.680838 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:17.680914 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 09:26:17.680955 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":69,"threads_started":1}
I20260926 09:26:17.681995 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:17.686586 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:17.683347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40241 (local address 127.0.0.1:33447)
0926 09:26:17.683383 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:17.683385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:17.685475 (+ 2090us) server_negotiation.cc:407] Connection header received
0926 09:26:17.685626 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:17.685627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:17.685633 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:17.685683 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:17.686339 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:17.686340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:17.686341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:17.686345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:17.686360 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:17.686364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:17.686420 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:17.686477 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 09:26:17.686492 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:26:17.692591 13849 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:17.692740 13849 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:17.700444 13849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6a4f4996bd747dbac40f16cc1244268: Generated new TSK 0
I20260926 09:26:17.700640 13849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:17.706918 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13769
2026-09-26T09:26:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:17 dist-test-slave-7zlw krb5kdc[13868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:17 dist-test-slave-7zlw krb5kdc[13868](info): set up 2 sockets
Sep 26 09:26:17 dist-test-slave-7zlw krb5kdc[13868](info): commencing operation
krb5kdc: starting...
W20260926 09:26:18.745489 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:18 dist-test-slave-7zlw krb5kdc[13868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:18.872987 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:18.957288 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:18.957511 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:18.957556 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:18.958812 13881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:18.958864 13881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:18.958878 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:18.958891 13881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:18.958904 13881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:18.960458 13881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:18.960731 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:18.960971 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:18.966174 13886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:18.966174 13887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:18.966204 13889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:18.966408 13881 server_base.cc:1034] running on GCE node
I20260926 09:26:18.966776 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:18.967051 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:18.968219 13881 hybrid_clock.cc:648] HybridClock initialized: now 1790414778968179 us; error 55 us; skew 500 ppm
Sep 26 09:26:18 dist-test-slave-7zlw krb5kdc[13868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:18.971201 13881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:18.971724 13881 webserver.cc:460] Webserver started at http://127.0.0.1:35157/ using document root <none> and password file <none>
I20260926 09:26:18.972000 13881 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:18.972057 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:18.972245 13881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:18.973587 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "f4374b5b73e5434bad21cdab53057164"
format_stamp: "Formatted at 2026-09-26 09:26:18 on dist-test-slave-7zlw"
server_key: "b8f2a93fcbb5a0024ce21e875a5de4e5"
server_key_iv: "0f32724013d9a39014c2c199c8d75110"
server_key_version: "encryptionkey@0"
I20260926 09:26:18.973992 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "f4374b5b73e5434bad21cdab53057164"
format_stamp: "Formatted at 2026-09-26 09:26:18 on dist-test-slave-7zlw"
server_key: "b8f2a93fcbb5a0024ce21e875a5de4e5"
server_key_iv: "0f32724013d9a39014c2c199c8d75110"
server_key_version: "encryptionkey@0"
I20260926 09:26:18.975481 13881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:18.976325 13896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:18.976529 13881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:18.976673 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "f4374b5b73e5434bad21cdab53057164"
format_stamp: "Formatted at 2026-09-26 09:26:18 on dist-test-slave-7zlw"
server_key: "b8f2a93fcbb5a0024ce21e875a5de4e5"
server_key_iv: "0f32724013d9a39014c2c199c8d75110"
server_key_version: "encryptionkey@0"
I20260926 09:26:18.976783 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:19.016593 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:19.019345 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:19.019510 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:19.024003 13881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46135
I20260926 09:26:19.024037 13948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46135 every 8 connection(s)
I20260926 09:26:19.024413 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:19.025470 13949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:19.027957 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: Bootstrap starting.
I20260926 09:26:19.028570 13949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:19.028903 13949 log.cc:824] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:19.029188 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13881
I20260926 09:26:19.029306 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:19.029500 25517 external_mini_cluster.cc:1428] Setting key 92d88315e19f8a2866c834ad7077cecf
I20260926 09:26:19.029790 13949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: No bootstrap required, opened a new log
I20260926 09:26:19.031924 13949 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260926 09:26:19.032115 13949 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:19.032188 13949 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4374b5b73e5434bad21cdab53057164, State: Initialized, Role: FOLLOWER
I20260926 09:26:19.032356 13949 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260926 09:26:19.032460 13949 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:19.032510 13949 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:19.032565 13949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:19.033723 13949 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260926 09:26:19.033860 13949 leader_election.cc:302] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4374b5b73e5434bad21cdab53057164; no voters:
I20260926 09:26:19.034093 13949 leader_election.cc:288] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:19.034132 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:19.034260 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [term 1 LEADER]: Becoming Leader. State: Replica: f4374b5b73e5434bad21cdab53057164, State: Running, Role: LEADER
I20260926 09:26:19.034402 13949 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:19.034416 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260926 09:26:19.036187 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4374b5b73e5434bad21cdab53057164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } } }
I20260926 09:26:19.036311 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:19.036190 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4374b5b73e5434bad21cdab53057164. Latest consensus state: current_term: 1 leader_uuid: "f4374b5b73e5434bad21cdab53057164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4374b5b73e5434bad21cdab53057164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } } }
I20260926 09:26:19.037936 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164 [sys.catalog]: This master's current role is: LEADER
Sep 26 09:26:19 dist-test-slave-7zlw krb5kdc[13868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260926 09:26:19.037160 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:19.038587 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:19.039026 13969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:19.039278 13969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:19.042158 13969 catalog_manager.cc:1347] Generated new cluster ID: 72085fd02e2f4c5fb1a160c58f52f979
I20260926 09:26:19.042218 13969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:19.043318 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:19.030093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:39213 (local address 127.0.0.1:46135)
0926 09:26:19.030300 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:19.030313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:19.030380 (+ 67us) server_negotiation.cc:407] Connection header received
0926 09:26:19.031868 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:19.031877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:19.031941 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:19.032111 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:19.032894 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:19.033909 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:19.035112 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:19.035842 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:19.035950 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:19.039057 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:19.039061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:19.039061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:19.039074 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:19.041170 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:19.041683 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:19.041687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:19.041687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:19.041821 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:19.042141 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:19.042142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:19.042143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:19.042465 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:19.042638 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:19.042792 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 09:26:19.042902 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260926 09:26:19.044400 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:19.053875 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:19.045994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:43101 (local address 127.0.0.1:46135)
0926 09:26:19.046043 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:19.046046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:19.046186 (+ 140us) server_negotiation.cc:407] Connection header received
0926 09:26:19.046284 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:19.046285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:19.046296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:19.046325 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:19.047041 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:19.047838 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:19.048887 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:19.049255 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:19.049303 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:19.050838 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:19.050840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:19.050840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:19.050847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:19.052474 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:19.052905 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:19.052906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:19.052906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:19.052961 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:19.053260 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:19.053261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:19.053261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:19.053390 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:19.053467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:19.053710 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:26:19.053777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 09:26:19.066725 13969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:19.067325 13969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:19.073503 13969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4374b5b73e5434bad21cdab53057164: Generated new TSK 0
I20260926 09:26:19.073717 13969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:19.082157 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13881
W20260926 09:26:19.087236 13866 connection.cc:535] client connection to 127.0.0.1:46135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46135): Connection reset by peer (error 104)
2026-09-26T09:26:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:19 dist-test-slave-7zlw krb5kdc[13980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:19 dist-test-slave-7zlw krb5kdc[13980](info): set up 2 sockets
Sep 26 09:26:19 dist-test-slave-7zlw krb5kdc[13980](info): commencing operation
krb5kdc: starting...
W20260926 09:26:20.113574 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:20 dist-test-slave-7zlw krb5kdc[13980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:20.240688 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:20.324460 13993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:20.324728 13993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:20.324752 13993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:20.326064 13993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:20.326115 13993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:20.326129 13993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:20.326143 13993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:20.326154 13993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:20.327723 13993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:20.327986 13993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:20.328222 13993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:20.333348 13998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:20.333339 13999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:20.333542 13993 server_base.cc:1034] running on GCE node
W20260926 09:26:20.333343 14001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:20.333995 13993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:20.334261 13993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:20.335441 13993 hybrid_clock.cc:648] HybridClock initialized: now 1790414780335432 us; error 44 us; skew 500 ppm
Sep 26 09:26:20 dist-test-slave-7zlw krb5kdc[13980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:20.338248 13993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:20.338686 13993 webserver.cc:460] Webserver started at http://127.0.0.1:35985/ using document root <none> and password file <none>
I20260926 09:26:20.338924 13993 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:20.338972 13993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:20.339126 13993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:20.340238 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "7695eed150eb4eedbeb616836d63b521"
format_stamp: "Formatted at 2026-09-26 09:26:20 on dist-test-slave-7zlw"
server_key: "438cca59de4f6473d074c7a6517112c3"
server_key_iv: "64131fa35d12968e0f669e6ee3bd744f"
server_key_version: "encryptionkey@0"
I20260926 09:26:20.340626 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "7695eed150eb4eedbeb616836d63b521"
format_stamp: "Formatted at 2026-09-26 09:26:20 on dist-test-slave-7zlw"
server_key: "438cca59de4f6473d074c7a6517112c3"
server_key_iv: "64131fa35d12968e0f669e6ee3bd744f"
server_key_version: "encryptionkey@0"
I20260926 09:26:20.342139 13993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:20.343050 14008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:20.343263 13993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:26:20.343382 13993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "7695eed150eb4eedbeb616836d63b521"
format_stamp: "Formatted at 2026-09-26 09:26:20 on dist-test-slave-7zlw"
server_key: "438cca59de4f6473d074c7a6517112c3"
server_key_iv: "64131fa35d12968e0f669e6ee3bd744f"
server_key_version: "encryptionkey@0"
I20260926 09:26:20.343461 13993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:20.357854 13993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:20.360641 13993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:20.360791 13993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:20.365063 13993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38955
I20260926 09:26:20.365109 14060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38955 every 8 connection(s)
I20260926 09:26:20.365497 13993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:20.366109 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 13993
I20260926 09:26:20.366245 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:20.366467 25517 external_mini_cluster.cc:1428] Setting key 69a6e073f4654e59fa5eed8c7b5b38e9
I20260926 09:26:20.366750 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:20.369959 14061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521: Bootstrap starting.
I20260926 09:26:20.370754 14061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:20.371104 14061 log.cc:824] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:20.371892 14061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521: No bootstrap required, opened a new log
I20260926 09:26:20.372469 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:20.367095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:44045 (local address 127.0.0.1:38955)
0926 09:26:20.367332 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:20.367341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:20.367367 (+ 26us) server_negotiation.cc:407] Connection header received
0926 09:26:20.367935 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:20.367941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:20.367989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:20.368142 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:20.368381 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:20.369387 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:20.370196 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:20.370809 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:20.370944 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:20.371666 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:20.371671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:20.371671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:20.371886 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:20.371926 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:20.371942 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:20.372048 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:20.372219 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 09:26:20.372278 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260926 09:26:20.373672 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:20.374281 14061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } }
I20260926 09:26:20.374413 14061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:20.374435 14061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7695eed150eb4eedbeb616836d63b521, State: Initialized, Role: FOLLOWER
I20260926 09:26:20.374580 14061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } }
I20260926 09:26:20.374662 14061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:20.374697 14061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:20.374750 14061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:20.375449 14061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } }
I20260926 09:26:20.375581 14061 leader_election.cc:302] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7695eed150eb4eedbeb616836d63b521; no voters:
I20260926 09:26:20.375744 14061 leader_election.cc:288] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:20.375782 14071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:20.375998 14061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:20.376024 14071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [term 1 LEADER]: Becoming Leader. State: Replica: 7695eed150eb4eedbeb616836d63b521, State: Running, Role: LEADER
I20260926 09:26:20.376113 14071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } }
I20260926 09:26:20.376518 14072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7695eed150eb4eedbeb616836d63b521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } } }
I20260926 09:26:20.376526 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7695eed150eb4eedbeb616836d63b521. Latest consensus state: current_term: 1 leader_uuid: "7695eed150eb4eedbeb616836d63b521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7695eed150eb4eedbeb616836d63b521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38955 } } }
I20260926 09:26:20.376657 14072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:20.376778 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:20.376988 14080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:20.377530 14080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:20.379182 14080 catalog_manager.cc:1347] Generated new cluster ID: ea48875483304c1387b005fc20a790fc
I20260926 09:26:20.379246 14080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:20.380640 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:20.375072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:57787 (local address 127.0.0.1:38955)
0926 09:26:20.375111 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:20.375112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:20.377406 (+ 2294us) server_negotiation.cc:407] Connection header received
0926 09:26:20.377484 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:20.377485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:20.377495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:20.377568 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:20.377749 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:20.378391 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:20.379352 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:20.379573 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:20.379655 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:20.380349 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:20.380351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:20.380351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:20.380356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:20.380371 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:20.380375 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:20.380431 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:20.380492 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:26:20.380501 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:26:20.390559 14080 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:20.391311 14080 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:20.395897 14080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7695eed150eb4eedbeb616836d63b521: Generated new TSK 0
I20260926 09:26:20.396103 14080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:20.399644 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 13993
W20260926 09:26:20.404564 13978 connection.cc:535] client connection to 127.0.0.1:38955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38955): Connection reset by peer (error 104)
2026-09-26T09:26:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:20 dist-test-slave-7zlw krb5kdc[14092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:20 dist-test-slave-7zlw krb5kdc[14092](info): set up 2 sockets
Sep 26 09:26:20 dist-test-slave-7zlw krb5kdc[14092](info): commencing operation
krb5kdc: starting...
W20260926 09:26:21.489348 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:21 dist-test-slave-7zlw krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:21.618373 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:21.699266 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:21.699506 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:21.699544 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:21.700696 14105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:21.700765 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:21.700790 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:21.700817 14105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:21.700850 14105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:21.702346 14105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:21.702620 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:21.702860 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:21.707996 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:21.708170 14105 server_base.cc:1034] running on GCE node
W20260926 09:26:21.708004 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:21.708117 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:21.708567 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:21.708868 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:21.710055 14105 hybrid_clock.cc:648] HybridClock initialized: now 1790414781710035 us; error 48 us; skew 500 ppm
Sep 26 09:26:21 dist-test-slave-7zlw krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:21.712702 14105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:21.713253 14105 webserver.cc:460] Webserver started at http://127.0.0.1:43681/ using document root <none> and password file <none>
I20260926 09:26:21.713500 14105 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:21.713567 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:21.713728 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:21.714782 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "7a87b5183f1e4ea287b94f9c6955336c"
format_stamp: "Formatted at 2026-09-26 09:26:21 on dist-test-slave-7zlw"
server_key: "146cd74775a8768abfccb81e1d8889db"
server_key_iv: "26f407d2ec4b47c8e213e7b163a0beea"
server_key_version: "encryptionkey@0"
I20260926 09:26:21.715138 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "7a87b5183f1e4ea287b94f9c6955336c"
format_stamp: "Formatted at 2026-09-26 09:26:21 on dist-test-slave-7zlw"
server_key: "146cd74775a8768abfccb81e1d8889db"
server_key_iv: "26f407d2ec4b47c8e213e7b163a0beea"
server_key_version: "encryptionkey@0"
I20260926 09:26:21.716512 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:21.717559 14120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:21.717768 14105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:26:21.717891 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "7a87b5183f1e4ea287b94f9c6955336c"
format_stamp: "Formatted at 2026-09-26 09:26:21 on dist-test-slave-7zlw"
server_key: "146cd74775a8768abfccb81e1d8889db"
server_key_iv: "26f407d2ec4b47c8e213e7b163a0beea"
server_key_version: "encryptionkey@0"
I20260926 09:26:21.717984 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:21.721508 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:21.724362 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:21.724573 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:21.729228 14105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43329
I20260926 09:26:21.729271 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43329 every 8 connection(s)
I20260926 09:26:21.729662 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:21.730727 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:21.733105 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c: Bootstrap starting.
I20260926 09:26:21.733531 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14105
I20260926 09:26:21.733667 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:21.733801 14173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:21.733866 25517 external_mini_cluster.cc:1428] Setting key 3e46fd6d5f825ca095e6923437a2a3f1
I20260926 09:26:21.734115 14173 log.cc:824] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:21.734928 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c: No bootstrap required, opened a new log
I20260926 09:26:21.736483 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:21.734416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:52051 (local address 127.0.0.1:43329)
0926 09:26:21.734670 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:21.734679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:21.734716 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:26:21.735220 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:21.735227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:21.735254 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:21.735370 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:21.736017 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:21.736020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:21.736021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:21.736024 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:21.736060 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:21.736073 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:21.736141 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:21.736224 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 09:26:21.736282 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260926 09:26:21.737483 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:21.737481 14173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260926 09:26:21.737677 14173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:21.737735 14173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a87b5183f1e4ea287b94f9c6955336c, State: Initialized, Role: FOLLOWER
I20260926 09:26:21.737860 14173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260926 09:26:21.737952 14173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:21.737995 14173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:21.738044 14173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:21.738623 14173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260926 09:26:21.738740 14173 leader_election.cc:302] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a87b5183f1e4ea287b94f9c6955336c; no voters:
I20260926 09:26:21.738921 14173 leader_election.cc:288] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:21.738958 14182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:21.739154 14182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [term 1 LEADER]: Becoming Leader. State: Replica: 7a87b5183f1e4ea287b94f9c6955336c, State: Running, Role: LEADER
I20260926 09:26:21.739185 14173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:21.739305 14182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260926 09:26:21.739753 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a87b5183f1e4ea287b94f9c6955336c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } } }
I20260926 09:26:21.739810 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a87b5183f1e4ea287b94f9c6955336c. Latest consensus state: current_term: 1 leader_uuid: "7a87b5183f1e4ea287b94f9c6955336c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a87b5183f1e4ea287b94f9c6955336c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } } }
I20260926 09:26:21.739867 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:21.739970 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:21.740183 14192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:21.740506 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:21.739029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:45943 (local address 127.0.0.1:43329)
0926 09:26:21.739079 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:21.739081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:21.739322 (+ 241us) server_negotiation.cc:407] Connection header received
0926 09:26:21.739424 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:21.739425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:21.739430 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:21.739458 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:21.740181 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:21.740182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:21.740183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:21.740196 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:21.740213 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:21.740218 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:21.740285 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:21.740403 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 09:26:21.740419 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:21.740787 14192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:21.742871 14192 catalog_manager.cc:1347] Generated new cluster ID: b141c84a930442b1bae8c46daa42197a
I20260926 09:26:21.742939 14192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:21.780412 14192 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:21.780622 14192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:21.786322 14192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a87b5183f1e4ea287b94f9c6955336c: Generated new TSK 0
I20260926 09:26:21.786544 14192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:21.795135 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14105
2026-09-26T09:26:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:21 dist-test-slave-7zlw krb5kdc[14204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:21 dist-test-slave-7zlw krb5kdc[14204](info): set up 2 sockets
Sep 26 09:26:21 dist-test-slave-7zlw krb5kdc[14204](info): commencing operation
krb5kdc: starting...
W20260926 09:26:22.833289 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:22 dist-test-slave-7zlw krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:22.960415 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:23.047776 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:23.047992 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:23.048017 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:23.049240 14217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:23.049288 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:23.049304 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:23.049316 14217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:23.049328 14217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:23.050872 14217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:23.051139 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:23.051369 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:23.056358 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:23.056492 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:23.056502 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:23.056771 14217 server_base.cc:1034] running on GCE node
I20260926 09:26:23.057005 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:23.057355 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:23.058553 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790414783058530 us; error 35 us; skew 500 ppm
Sep 26 09:26:23 dist-test-slave-7zlw krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:23.068285 14217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:23.068878 14217 webserver.cc:460] Webserver started at http://127.0.0.1:36745/ using document root <none> and password file <none>
I20260926 09:26:23.069211 14217 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:23.069290 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:23.069461 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:23.070712 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "02aeb8d65c9c4d389fd95e05120b2860"
format_stamp: "Formatted at 2026-09-26 09:26:23 on dist-test-slave-7zlw"
server_key: "384323faf2405aaf67e467e9d3145663"
server_key_iv: "3bb08b544d933b1beb8518fb7ea3484d"
server_key_version: "encryptionkey@0"
I20260926 09:26:23.071110 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "02aeb8d65c9c4d389fd95e05120b2860"
format_stamp: "Formatted at 2026-09-26 09:26:23 on dist-test-slave-7zlw"
server_key: "384323faf2405aaf67e467e9d3145663"
server_key_iv: "3bb08b544d933b1beb8518fb7ea3484d"
server_key_version: "encryptionkey@0"
I20260926 09:26:23.072603 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:23.073578 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:23.073793 14217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 09:26:23.073923 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "02aeb8d65c9c4d389fd95e05120b2860"
format_stamp: "Formatted at 2026-09-26 09:26:23 on dist-test-slave-7zlw"
server_key: "384323faf2405aaf67e467e9d3145663"
server_key_iv: "3bb08b544d933b1beb8518fb7ea3484d"
server_key_version: "encryptionkey@0"
I20260926 09:26:23.074023 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:23.082873 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:23.085407 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:23.085600 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:23.090197 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45033
I20260926 09:26:23.090229 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45033 every 8 connection(s)
I20260926 09:26:23.090626 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:23.091492 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:23.093852 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860: Bootstrap starting.
I20260926 09:26:23.094408 14285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:23.094640 14285 log.cc:824] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:23.095363 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860: No bootstrap required, opened a new log
I20260926 09:26:23.095397 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14217
I20260926 09:26:23.095522 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:23.095712 25517 external_mini_cluster.cc:1428] Setting key 126909d0d86a70854dce4dc3f93e7c49
I20260926 09:26:23.097393 14285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260926 09:26:23.097524 14285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:23.097546 14285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02aeb8d65c9c4d389fd95e05120b2860, State: Initialized, Role: FOLLOWER
I20260926 09:26:23.097679 14285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260926 09:26:23.097790 14285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:23.097834 14285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:23.097908 14285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:23.098814 14285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260926 09:26:23.098934 14285 leader_election.cc:302] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02aeb8d65c9c4d389fd95e05120b2860; no voters:
I20260926 09:26:23.099133 14285 leader_election.cc:288] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:23.099150 14290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:23.099268 14290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [term 1 LEADER]: Becoming Leader. State: Replica: 02aeb8d65c9c4d389fd95e05120b2860, State: Running, Role: LEADER
I20260926 09:26:23.099406 14285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:23.099406 14290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260926 09:26:23.099891 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02aeb8d65c9c4d389fd95e05120b2860. Latest consensus state: current_term: 1 leader_uuid: "02aeb8d65c9c4d389fd95e05120b2860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260926 09:26:23.099984 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:23.100203 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02aeb8d65c9c4d389fd95e05120b2860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aeb8d65c9c4d389fd95e05120b2860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260926 09:26:23.100323 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:23.100616 14297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:23.101156 14297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:23.102907 14297 catalog_manager.cc:1347] Generated new cluster ID: 0232faac1ee44d249aa9c56cbe69071c
I20260926 09:26:23.103083 14297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:23 dist-test-slave-7zlw krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 09:26:23.108083 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:23.096260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:33233 (local address 127.0.0.1:45033)
0926 09:26:23.096453 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:23.096463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:23.096498 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:26:23.097381 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:23.097389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:23.097442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:23.097683 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:23.098379 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:23.099477 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:23.100538 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:23.101167 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:23.101701 (+ 534us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:23.104136 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:23.104140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:23.104140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:23.104152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:23.106169 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:23.106595 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:23.106597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:23.106597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:23.106712 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:23.106919 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:23.106919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:23.106920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:23.107195 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:23.107719 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:23.107796 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 09:26:23.107890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260926 09:26:23.109453 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:23.117309 14297 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:23.117982 14297 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:23.118399 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:23.110962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:51607 (local address 127.0.0.1:45033)
0926 09:26:23.111003 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:23.111005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:23.111165 (+ 160us) server_negotiation.cc:407] Connection header received
0926 09:26:23.111229 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:23.111230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:23.111240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:23.111292 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:23.111902 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:23.112655 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:23.113714 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:23.114025 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:23.114082 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:23.115517 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:23.115518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:23.115519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:23.115525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:23.117154 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:23.117564 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:23.117565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:23.117566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:23.117620 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:23.117856 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:23.117857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:23.117857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:23.117967 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:23.118076 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:23.118224 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 09:26:23.118289 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:23.121913 14297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02aeb8d65c9c4d389fd95e05120b2860: Generated new TSK 0
I20260926 09:26:23.122087 14297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:23.125907 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14217
W20260926 09:26:23.130465 14202 connection.cc:535] client connection to 127.0.0.1:45033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45033): Connection reset by peer (error 104)
2026-09-26T09:26:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:23 dist-test-slave-7zlw krb5kdc[14316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:23 dist-test-slave-7zlw krb5kdc[14316](info): set up 2 sockets
Sep 26 09:26:23 dist-test-slave-7zlw krb5kdc[14316](info): commencing operation
krb5kdc: starting...
W20260926 09:26:24.169450 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:24 dist-test-slave-7zlw krb5kdc[14316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:24.305254 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:24.388712 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:24.388979 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:24.389016 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:24.390237 14329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:24.390313 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:24.390339 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:24.390372 14329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:24.390395 14329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:24.391906 14329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:24.392195 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:24.392432 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:24.397562 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:24.397622 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:24.397668 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:24.397827 14329 server_base.cc:1034] running on GCE node
I20260926 09:26:24.398046 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:24.398300 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:24.399446 14329 hybrid_clock.cc:648] HybridClock initialized: now 1790414784399420 us; error 41 us; skew 500 ppm
Sep 26 09:26:24 dist-test-slave-7zlw krb5kdc[14316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:24.402163 14329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:24.402742 14329 webserver.cc:460] Webserver started at http://127.0.0.1:41103/ using document root <none> and password file <none>
I20260926 09:26:24.402983 14329 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:24.403061 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:24.403227 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:24.404336 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "93dfc2a5a665461bb836bc021bfaf32a"
format_stamp: "Formatted at 2026-09-26 09:26:24 on dist-test-slave-7zlw"
server_key: "f40f1741a09c0e2d0ca8f160d08edb2e"
server_key_iv: "3e6e748aa2c9ab6bbef8541a6704079d"
server_key_version: "encryptionkey@0"
I20260926 09:26:24.404724 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "93dfc2a5a665461bb836bc021bfaf32a"
format_stamp: "Formatted at 2026-09-26 09:26:24 on dist-test-slave-7zlw"
server_key: "f40f1741a09c0e2d0ca8f160d08edb2e"
server_key_iv: "3e6e748aa2c9ab6bbef8541a6704079d"
server_key_version: "encryptionkey@0"
I20260926 09:26:24.406243 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:24.407176 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:24.407434 14329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:24.407548 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "93dfc2a5a665461bb836bc021bfaf32a"
format_stamp: "Formatted at 2026-09-26 09:26:24 on dist-test-slave-7zlw"
server_key: "f40f1741a09c0e2d0ca8f160d08edb2e"
server_key_iv: "3e6e748aa2c9ab6bbef8541a6704079d"
server_key_version: "encryptionkey@0"
I20260926 09:26:24.407632 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:24.419963 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:24.422703 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:24.422905 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:24.427731 14329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36143
I20260926 09:26:24.427793 14396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36143 every 8 connection(s)
I20260926 09:26:24.428194 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:24.429172 14397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:24.430655 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14329
I20260926 09:26:24.430819 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:24.431005 25517 external_mini_cluster.cc:1428] Setting key de253d6b8ab624072682db4afaa4f104
I20260926 09:26:24.431933 14397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a: Bootstrap starting.
I20260926 09:26:24.432618 14397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:24.432964 14397 log.cc:824] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:24.433800 14397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a: No bootstrap required, opened a new log
I20260926 09:26:24.436030 14397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 09:26:24.436229 14397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:24.436266 14397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dfc2a5a665461bb836bc021bfaf32a, State: Initialized, Role: FOLLOWER
I20260926 09:26:24.436393 14397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 09:26:24.436479 14397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:24.436515 14397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:24.436546 14397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:24.437184 14397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 09:26:24.437302 14397 leader_election.cc:302] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93dfc2a5a665461bb836bc021bfaf32a; no voters:
I20260926 09:26:24.437489 14397 leader_election.cc:288] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:24.437515 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:24.431641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:40133 (local address 127.0.0.1:36143)
0926 09:26:24.431838 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:24.431845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:24.431872 (+ 27us) server_negotiation.cc:407] Connection header received
0926 09:26:24.432345 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:24.432352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:24.432405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:24.432527 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:24.432823 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:24.434202 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:24.435066 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:24.435767 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:24.435879 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:24.436717 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:24.436720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:24.436721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:24.436904 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:24.436939 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:24.436949 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:24.437140 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:24.437255 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 09:26:24.437291 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260926 09:26:24.437518 14402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:24.437719 14402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [term 1 LEADER]: Becoming Leader. State: Replica: 93dfc2a5a665461bb836bc021bfaf32a, State: Running, Role: LEADER
I20260926 09:26:24.437759 14397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:24.437798 14402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260926 09:26:24.438146 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93dfc2a5a665461bb836bc021bfaf32a. Latest consensus state: current_term: 1 leader_uuid: "93dfc2a5a665461bb836bc021bfaf32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260926 09:26:24.438251 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:24.438158 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93dfc2a5a665461bb836bc021bfaf32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93dfc2a5a665461bb836bc021bfaf32a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260926 09:26:24.438441 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:24.438627 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:24.439332 14409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:24.439850 14409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:24.441642 14409 catalog_manager.cc:1347] Generated new cluster ID: 5fb4ee5b4253416aa57fef6f1cf4287e
I20260926 09:26:24.441697 14409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:24.443975 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:24.440275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:35091 (local address 127.0.0.1:36143)
0926 09:26:24.440310 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:24.440312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:24.440439 (+ 127us) server_negotiation.cc:407] Connection header received
0926 09:26:24.440511 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:24.440512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:24.440523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:24.440570 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:24.440717 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:24.441361 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:24.442412 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:24.442702 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:24.442774 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:24.443660 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:24.443661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:24.443662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:24.443667 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:24.443684 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:24.443689 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:24.443727 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:24.443884 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 09:26:24.443897 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:26:24.474666 14409 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:24.475284 14409 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:24.480660 14409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93dfc2a5a665461bb836bc021bfaf32a: Generated new TSK 0
I20260926 09:26:24.480907 14409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:24.490180 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14329
W20260926 09:26:24.495441 14314 connection.cc:535] client connection to 127.0.0.1:36143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36143): Connection reset by peer (error 104)
2026-09-26T09:26:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:24 dist-test-slave-7zlw krb5kdc[14428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:24 dist-test-slave-7zlw krb5kdc[14428](info): set up 2 sockets
Sep 26 09:26:24 dist-test-slave-7zlw krb5kdc[14428](info): commencing operation
krb5kdc: starting...
W20260926 09:26:25.561386 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:25 dist-test-slave-7zlw krb5kdc[14428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 09:26:25.689661 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:25.772037 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:25.772291 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:25.772316 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:25.773561 14441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:25.773603 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:25.773618 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:25.773630 14441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:25.773644 14441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:25.775148 14441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:25.775404 14441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:25.775676 14441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:25.781275 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:25.781293 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:25.781283 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:25.781656 14441 server_base.cc:1034] running on GCE node
I20260926 09:26:25.781926 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:25.782244 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:25.783425 14441 hybrid_clock.cc:648] HybridClock initialized: now 1790414785783420 us; error 40 us; skew 500 ppm
Sep 26 09:26:25 dist-test-slave-7zlw krb5kdc[14428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:25.786285 14441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 09:26:25.786849 14441 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260926 09:26:25.787139 14441 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:25.787218 14441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:25.787384 14441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:25.788569 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "965097325b544a449d0fd3e3c9ef9bdd"
format_stamp: "Formatted at 2026-09-26 09:26:25 on dist-test-slave-7zlw"
server_key: "42164c0f56858834e2772ede3fbf53f9"
server_key_iv: "c95664a384bcbc368a03e0077e86264a"
server_key_version: "encryptionkey@0"
I20260926 09:26:25.788975 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "965097325b544a449d0fd3e3c9ef9bdd"
format_stamp: "Formatted at 2026-09-26 09:26:25 on dist-test-slave-7zlw"
server_key: "42164c0f56858834e2772ede3fbf53f9"
server_key_iv: "c95664a384bcbc368a03e0077e86264a"
server_key_version: "encryptionkey@0"
I20260926 09:26:25.790637 14441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:25.791536 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:25.791855 14441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:25.791991 14441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "965097325b544a449d0fd3e3c9ef9bdd"
format_stamp: "Formatted at 2026-09-26 09:26:25 on dist-test-slave-7zlw"
server_key: "42164c0f56858834e2772ede3fbf53f9"
server_key_iv: "c95664a384bcbc368a03e0077e86264a"
server_key_version: "encryptionkey@0"
I20260926 09:26:25.792106 14441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:25.796139 14441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:25.798568 14441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:25.798758 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:25.803164 14441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46357
I20260926 09:26:25.803244 14508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46357 every 8 connection(s)
I20260926 09:26:25.803610 14441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:25.804677 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:25.804970 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14441
I20260926 09:26:25.805126 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:25.805342 25517 external_mini_cluster.cc:1428] Setting key 683c66257cafa21ec85d04f4159579d3
I20260926 09:26:25.807897 14509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd: Bootstrap starting.
I20260926 09:26:25.808151 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:25.805967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:53763 (local address 127.0.0.1:46357)
0926 09:26:25.806213 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:25.806223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:25.806258 (+ 35us) server_negotiation.cc:407] Connection header received
0926 09:26:25.806793 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:25.806801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:25.806834 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:25.806989 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:25.807720 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:25.807723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:25.807724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:25.807728 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:25.807782 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:25.807796 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:25.807871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:25.807930 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 09:26:25.807973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260926 09:26:25.808631 14509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:25.808928 14509 log.cc:824] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:25.809208 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:25.809669 14509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd: No bootstrap required, opened a new log
I20260926 09:26:25.811730 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:25.810712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:48821 (local address 127.0.0.1:46357)
0926 09:26:25.810752 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:25.810754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:25.810767 (+ 13us) server_negotiation.cc:407] Connection header received
0926 09:26:25.810831 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:25.810832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:25.810837 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:25.810902 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:25.811483 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:25.811484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:25.811485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:25.811489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:25.811507 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:25.811511 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:25.811599 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:25.811635 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 09:26:25.811651 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:26:25.812120 14509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } }
I20260926 09:26:25.812288 14509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:25.812328 14509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965097325b544a449d0fd3e3c9ef9bdd, State: Initialized, Role: FOLLOWER
I20260926 09:26:25.812431 14509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } }
I20260926 09:26:25.812497 14509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:25.812518 14509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:25.812546 14509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:25.813117 14509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } }
I20260926 09:26:25.813203 14509 leader_election.cc:302] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965097325b544a449d0fd3e3c9ef9bdd; no voters:
I20260926 09:26:25.813318 14509 leader_election.cc:288] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:25.813370 14519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:25.813505 14519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [term 1 LEADER]: Becoming Leader. State: Replica: 965097325b544a449d0fd3e3c9ef9bdd, State: Running, Role: LEADER
I20260926 09:26:25.813563 14509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:25.813629 14519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } }
I20260926 09:26:25.814071 14520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "965097325b544a449d0fd3e3c9ef9bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } } }
I20260926 09:26:25.814118 14521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965097325b544a449d0fd3e3c9ef9bdd. Latest consensus state: current_term: 1 leader_uuid: "965097325b544a449d0fd3e3c9ef9bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965097325b544a449d0fd3e3c9ef9bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46357 } } }
I20260926 09:26:25.814236 14520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:25.814251 14521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:25.814491 14528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:25.814985 14528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:25.818197 14528 catalog_manager.cc:1347] Generated new cluster ID: 2a2415acad69403b8d0f0261b5d904e0
I20260926 09:26:25.818259 14528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:25.826527 14528 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:25.826715 14528 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:25.837677 14528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965097325b544a449d0fd3e3c9ef9bdd: Generated new TSK 0
I20260926 09:26:25.837899 14528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:25.842815 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14441
2026-09-26T09:26:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:25 dist-test-slave-7zlw krb5kdc[14540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:25 dist-test-slave-7zlw krb5kdc[14540](info): set up 2 sockets
Sep 26 09:26:25 dist-test-slave-7zlw krb5kdc[14540](info): commencing operation
krb5kdc: starting...
W20260926 09:26:26.897856 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:26 dist-test-slave-7zlw krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:27.029311 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33901
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:27.112075 14553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:27.112295 14553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:27.112319 14553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:27.113495 14553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:27.113540 14553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:27.113555 14553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:27.113567 14553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:27.113579 14553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:27.115257 14553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:27.115470 14553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:27.115664 14553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:27.120535 14561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:27.120695 14558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:27.120846 14559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:27.120715 14553 server_base.cc:1034] running on GCE node
I20260926 09:26:27.121218 14553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:27.121534 14553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:27.122731 14553 hybrid_clock.cc:648] HybridClock initialized: now 1790414787122669 us; error 40 us; skew 500 ppm
Sep 26 09:26:27 dist-test-slave-7zlw krb5kdc[14540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414787, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:27.125381 14553 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:27.125854 14553 webserver.cc:460] Webserver started at http://127.24.235.126:40635/ using document root <none> and password file <none>
I20260926 09:26:27.126085 14553 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:27.126135 14553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:27.126300 14553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:27.127362 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "8b615c4295684aeda585fb42175ceabd"
format_stamp: "Formatted at 2026-09-26 09:26:27 on dist-test-slave-7zlw"
server_key: "4df15f842a55fea32cd2580ece2ccdd8"
server_key_iv: "c67a4902df9a82c43ac641ee8b96ca89"
server_key_version: "encryptionkey@0"
I20260926 09:26:27.127694 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "8b615c4295684aeda585fb42175ceabd"
format_stamp: "Formatted at 2026-09-26 09:26:27 on dist-test-slave-7zlw"
server_key: "4df15f842a55fea32cd2580ece2ccdd8"
server_key_iv: "c67a4902df9a82c43ac641ee8b96ca89"
server_key_version: "encryptionkey@0"
I20260926 09:26:27.129163 14553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:27.129978 14568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:27.130193 14553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:27.130304 14553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "8b615c4295684aeda585fb42175ceabd"
format_stamp: "Formatted at 2026-09-26 09:26:27 on dist-test-slave-7zlw"
server_key: "4df15f842a55fea32cd2580ece2ccdd8"
server_key_iv: "c67a4902df9a82c43ac641ee8b96ca89"
server_key_version: "encryptionkey@0"
I20260926 09:26:27.130381 14553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:27.143605 14553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:27.146026 14553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:27.146199 14553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:27.150489 14553 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33901
I20260926 09:26:27.150525 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33901 every 8 connection(s)
I20260926 09:26:27.150939 14553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:27.151924 14621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:27.154268 14621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd: Bootstrap starting.
I20260926 09:26:27.154654 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14553
I20260926 09:26:27.154778 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:27.154997 25517 external_mini_cluster.cc:1428] Setting key 67db75ae007fd48906f87224e406e7f2
I20260926 09:26:27.155004 14621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:27.155323 14621 log.cc:824] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:27.156116 14621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd: No bootstrap required, opened a new log
I20260926 09:26:27.158432 14621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } }
I20260926 09:26:27.158638 14621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:27.158679 14621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b615c4295684aeda585fb42175ceabd, State: Initialized, Role: FOLLOWER
I20260926 09:26:27.158804 14621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } }
I20260926 09:26:27.158886 14621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:27.158915 14621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:27.158958 14621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:27.159633 14621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } }
I20260926 09:26:27.159724 14621 leader_election.cc:302] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b615c4295684aeda585fb42175ceabd; no voters:
I20260926 09:26:27.159930 14621 leader_election.cc:288] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:27.159981 14626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:27.160182 14626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [term 1 LEADER]: Becoming Leader. State: Replica: 8b615c4295684aeda585fb42175ceabd, State: Running, Role: LEADER
I20260926 09:26:27.160211 14621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:27.160279 14626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } }
I20260926 09:26:27.160681 14626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b615c4295684aeda585fb42175ceabd. Latest consensus state: current_term: 1 leader_uuid: "8b615c4295684aeda585fb42175ceabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } } }
I20260926 09:26:27.160766 14626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:27.160847 14627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b615c4295684aeda585fb42175ceabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b615c4295684aeda585fb42175ceabd" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33901 } } }
I20260926 09:26:27.161150 14627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:27.161326 14635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:27.161917 14635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 09:26:27 dist-test-slave-7zlw krb5kdc[14540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:27.163762 14635 catalog_manager.cc:1347] Generated new cluster ID: 569a60ce842c4e2b8e92f17e0ba41ad6
I20260926 09:26:27.163812 14635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:27.168354 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:27.155552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34493 (local address 127.24.235.126:33901)
0926 09:26:27.155756 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:27.155766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:27.155805 (+ 39us) server_negotiation.cc:407] Connection header received
0926 09:26:27.156367 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:27.156375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:27.156428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:27.156552 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:27.157425 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:27.158395 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:27.159467 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:27.160022 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:27.160139 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:27.163316 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:27.163320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:27.163321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:27.163333 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:27.166555 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:27.167014 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:27.167018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:27.167019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:27.167121 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:27.167383 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:27.167384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:27.167385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:27.167650 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:27.167844 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:27.167986 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 09:26:27.168094 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260926 09:26:27.169478 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:27.173743 14635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:27.174489 14635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:27.178798 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:27.170959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37953 (local address 127.24.235.126:33901)
0926 09:26:27.171003 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:27.171005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:27.171159 (+ 154us) server_negotiation.cc:407] Connection header received
0926 09:26:27.171257 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:27.171257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:27.171269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:27.171330 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:27.171901 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:27.172597 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:27.173538 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:27.173830 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:27.173909 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:27.175402 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:27.175404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:27.175404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:27.175412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:27.177275 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:27.177742 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:27.177743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:27.177743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:27.177806 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:27.178089 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:27.178090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:27.178090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:27.178209 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:27.178322 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:27.178629 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 09:26:27.178707 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":22}
I20260926 09:26:27.178875 14635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b615c4295684aeda585fb42175ceabd: Generated new TSK 0
I20260926 09:26:27.179203 14635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:27.183441 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14553
W20260926 09:26:27.188323 14538 connection.cc:535] client connection to 127.24.235.126:33901 recv error: Network error: failed to read from TLS socket (remote: 127.24.235.126:33901): Connection reset by peer (error 104)
2026-09-26T09:26:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:27 dist-test-slave-7zlw krb5kdc[14652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:27 dist-test-slave-7zlw krb5kdc[14652](info): set up 2 sockets
Sep 26 09:26:27 dist-test-slave-7zlw krb5kdc[14652](info): commencing operation
krb5kdc: starting...
W20260926 09:26:28.229485 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:28 dist-test-slave-7zlw krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:28.361073 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33179
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:33179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:28.443907 14665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:28.444152 14665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:28.444191 14665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:28.445369 14665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:28.445437 14665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:28.445461 14665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:28.445482 14665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:28.445509 14665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:28.447001 14665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:33179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:33179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:28.447280 14665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:28.447511 14665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:28.452849 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:28.452920 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:28.452859 14670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:28.452903 14665 server_base.cc:1034] running on GCE node
I20260926 09:26:28.453408 14665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:28.453715 14665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:28.454882 14665 hybrid_clock.cc:648] HybridClock initialized: now 1790414788454868 us; error 46 us; skew 500 ppm
Sep 26 09:26:28 dist-test-slave-7zlw krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414788, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:28.465556 14665 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:28.466099 14665 webserver.cc:460] Webserver started at http://127.24.235.126:37171/ using document root <none> and password file <none>
I20260926 09:26:28.466367 14665 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:28.466416 14665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:28.466606 14665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:28.467746 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "6f3aa79780f94dbc9f8ac7c3892b12af"
format_stamp: "Formatted at 2026-09-26 09:26:28 on dist-test-slave-7zlw"
server_key: "4d8ee654a19b8bccba058d2748853904"
server_key_iv: "66c9aa81b5de564f675737baa2b72d77"
server_key_version: "encryptionkey@0"
I20260926 09:26:28.468113 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "6f3aa79780f94dbc9f8ac7c3892b12af"
format_stamp: "Formatted at 2026-09-26 09:26:28 on dist-test-slave-7zlw"
server_key: "4d8ee654a19b8bccba058d2748853904"
server_key_iv: "66c9aa81b5de564f675737baa2b72d77"
server_key_version: "encryptionkey@0"
I20260926 09:26:28.469717 14665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:28.470551 14680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:28.470774 14665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:28.470865 14665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "6f3aa79780f94dbc9f8ac7c3892b12af"
format_stamp: "Formatted at 2026-09-26 09:26:28 on dist-test-slave-7zlw"
server_key: "4d8ee654a19b8bccba058d2748853904"
server_key_iv: "66c9aa81b5de564f675737baa2b72d77"
server_key_version: "encryptionkey@0"
I20260926 09:26:28.470922 14665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:28.490706 14665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:28.493288 14665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:28.493413 14665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:28.497529 14665 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:33179
I20260926 09:26:28.497568 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:33179 every 8 connection(s)
I20260926 09:26:28.497948 14665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:28.498893 14733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:28.501286 14733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af: Bootstrap starting.
I20260926 09:26:28.501833 14733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:28.502084 14733 log.cc:824] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:28.502813 14733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af: No bootstrap required, opened a new log
I20260926 09:26:28.504428 14733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } }
I20260926 09:26:28.504572 14733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:28.504594 14733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3aa79780f94dbc9f8ac7c3892b12af, State: Initialized, Role: FOLLOWER
I20260926 09:26:28.504750 14733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } }
I20260926 09:26:28.504832 14733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:28.504854 14733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:28.504884 14733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:28.505772 14733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } }
I20260926 09:26:28.505859 14733 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3aa79780f94dbc9f8ac7c3892b12af; no voters:
I20260926 09:26:28.505987 14733 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:28.506043 14736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:28.506227 14736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [term 1 LEADER]: Becoming Leader. State: Replica: 6f3aa79780f94dbc9f8ac7c3892b12af, State: Running, Role: LEADER
I20260926 09:26:28.506319 14733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:28.506354 14736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } }
I20260926 09:26:28.506750 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3aa79780f94dbc9f8ac7c3892b12af. Latest consensus state: current_term: 1 leader_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } } }
I20260926 09:26:28.506739 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3aa79780f94dbc9f8ac7c3892b12af" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 33179 } } }
I20260926 09:26:28.506834 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:28.506841 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:28.507040 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14665
I20260926 09:26:28.507079 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:28.507155 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:28.507371 25517 external_mini_cluster.cc:1428] Setting key 67a4cc7e8bb1a1e6902fa70d62af132e
I20260926 09:26:28.507663 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:28.509388 14742 catalog_manager.cc:1347] Generated new cluster ID: 2fa70298034e409b861c4886e085542e
I20260926 09:26:28.509442 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:28.513196 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:28.508020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44653 (local address 127.24.235.126:33179)
0926 09:26:28.508562 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:28.508573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:28.508610 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:26:28.508750 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:28.508757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:28.508809 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:28.509210 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:28.509311 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:28.510280 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:28.511097 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:28.511529 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:28.511647 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:28.512411 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:28.512415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:28.512416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:28.512597 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:28.512628 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:28.512639 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:28.512739 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:28.512819 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 09:26:28.512945 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":94,"threads_started":1}
I20260926 09:26:28.514067 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:28.522549 14742 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:28.523101 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:28.524567 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:28.515447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42161 (local address 127.24.235.126:33179)
0926 09:26:28.515483 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:28.515484 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:28.521231 (+ 5747us) server_negotiation.cc:407] Connection header received
0926 09:26:28.521324 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:28.521325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:28.521336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:28.521390 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:28.521531 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:28.522098 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:28.523169 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:28.523454 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:28.523522 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:28.524306 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:28.524308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:28.524308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:28.524313 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:28.524328 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:28.524333 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:28.524389 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:28.524480 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 09:26:28.524492 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":18}
I20260926 09:26:28.526651 14742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3aa79780f94dbc9f8ac7c3892b12af: Generated new TSK 0
I20260926 09:26:28.526844 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:28.532145 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14665
W20260926 09:26:28.536739 14650 connection.cc:535] client connection to 127.24.235.126:33179 recv error: Network error: failed to read from TLS socket (remote: 127.24.235.126:33179): Connection reset by peer (error 104)
2026-09-26T09:26:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:28 dist-test-slave-7zlw krb5kdc[14764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:28 dist-test-slave-7zlw krb5kdc[14764](info): set up 2 sockets
Sep 26 09:26:28 dist-test-slave-7zlw krb5kdc[14764](info): commencing operation
krb5kdc: starting...
W20260926 09:26:29.581442 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:29 dist-test-slave-7zlw krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:29.710908 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:40033
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:29.797088 14777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:29.797369 14777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:29.797415 14777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:29.798664 14777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:29.798744 14777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:29.798769 14777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:29.798789 14777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:29.798820 14777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:29.800352 14777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:40033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:29.800658 14777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:29.800908 14777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:29.806247 14783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:29.806218 14782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:29.806249 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:29.806376 14777 server_base.cc:1034] running on GCE node
I20260926 09:26:29.806696 14777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:29.807018 14777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:29.808225 14777 hybrid_clock.cc:648] HybridClock initialized: now 1790414789808203 us; error 39 us; skew 500 ppm
Sep 26 09:26:29 dist-test-slave-7zlw krb5kdc[14764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414789, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:29.810876 14777 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:29.811358 14777 webserver.cc:460] Webserver started at http://127.24.235.126:33519/ using document root <none> and password file <none>
I20260926 09:26:29.811590 14777 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:29.811636 14777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:29.811774 14777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:29.812847 14777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "ae423097617e4471a8611619875a1e3c"
format_stamp: "Formatted at 2026-09-26 09:26:29 on dist-test-slave-7zlw"
server_key: "524988dc8318b4c5fd617730b7f76c35"
server_key_iv: "1b6509feb892af812b2e35d5cd0e8bc9"
server_key_version: "encryptionkey@0"
I20260926 09:26:29.813264 14777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "ae423097617e4471a8611619875a1e3c"
format_stamp: "Formatted at 2026-09-26 09:26:29 on dist-test-slave-7zlw"
server_key: "524988dc8318b4c5fd617730b7f76c35"
server_key_iv: "1b6509feb892af812b2e35d5cd0e8bc9"
server_key_version: "encryptionkey@0"
I20260926 09:26:29.814702 14777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:29.815534 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:29.815908 14777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:29.816027 14777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "ae423097617e4471a8611619875a1e3c"
format_stamp: "Formatted at 2026-09-26 09:26:29 on dist-test-slave-7zlw"
server_key: "524988dc8318b4c5fd617730b7f76c35"
server_key_iv: "1b6509feb892af812b2e35d5cd0e8bc9"
server_key_version: "encryptionkey@0"
I20260926 09:26:29.816116 14777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:29.819768 14777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:29.822386 14777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:29.822557 14777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:29.827296 14777 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:40033
I20260926 09:26:29.827340 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:40033 every 8 connection(s)
I20260926 09:26:29.827720 14777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:29.828752 14845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:29.831292 14845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c: Bootstrap starting.
I20260926 09:26:29.831910 14845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:29.832216 14845 log.cc:824] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:29.833072 14845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c: No bootstrap required, opened a new log
I20260926 09:26:29.834745 14845 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } }
I20260926 09:26:29.834892 14845 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:29.834915 14845 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae423097617e4471a8611619875a1e3c, State: Initialized, Role: FOLLOWER
I20260926 09:26:29.835079 14845 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } }
I20260926 09:26:29.835179 14845 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:29.835224 14845 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:29.835286 14845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:29.836179 14845 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } }
I20260926 09:26:29.836272 14845 leader_election.cc:302] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae423097617e4471a8611619875a1e3c; no voters:
I20260926 09:26:29.836496 14845 leader_election.cc:288] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:29.836617 14848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:29.836658 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14777
I20260926 09:26:29.836776 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:29.836807 14845 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:29.836836 14848 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [term 1 LEADER]: Becoming Leader. State: Replica: ae423097617e4471a8611619875a1e3c, State: Running, Role: LEADER
I20260926 09:26:29.836930 14848 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } }
I20260926 09:26:29.836961 25517 external_mini_cluster.cc:1428] Setting key 7863a2f6a9329eefd74b5d1a9ddd461f
I20260926 09:26:29.837376 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae423097617e4471a8611619875a1e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } } }
I20260926 09:26:29.837380 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae423097617e4471a8611619875a1e3c. Latest consensus state: current_term: 1 leader_uuid: "ae423097617e4471a8611619875a1e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae423097617e4471a8611619875a1e3c" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 40033 } } }
I20260926 09:26:29.837488 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:29.837492 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:29.838322 14858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:29.838904 14858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:29.840317 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:29.837670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40493 (local address 127.24.235.126:40033)
0926 09:26:29.837839 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:29.837850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:29.838071 (+ 221us) server_negotiation.cc:407] Connection header received
0926 09:26:29.838218 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:29.838228 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:29.838264 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:29.838413 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:29.839456 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:29.839459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:29.839460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:29.839464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:29.839507 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:29.839522 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:29.839594 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:29.840078 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 09:26:29.840124 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 09:26:29.840619 14858 catalog_manager.cc:1347] Generated new cluster ID: 27ace73bd24f4dc0965a65a1c551d0f9
I20260926 09:26:29.840672 14858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:29.841400 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:29.844430 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:29.843111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46409 (local address 127.24.235.126:40033)
0926 09:26:29.843154 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:29.843156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:29.843316 (+ 160us) server_negotiation.cc:407] Connection header received
0926 09:26:29.843418 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:29.843419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:29.843424 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:29.843454 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:29.844189 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:29.844190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:29.844190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:29.844195 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:29.844210 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:29.844215 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:29.844245 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:29.844352 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 09:26:29.844366 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:26:29.847144 14858 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:29.847303 14858 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:29.850886 14858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae423097617e4471a8611619875a1e3c: Generated new TSK 0
I20260926 09:26:29.851082 14858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:29.855464 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14777
2026-09-26T09:26:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:29 dist-test-slave-7zlw krb5kdc[14876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:29 dist-test-slave-7zlw krb5kdc[14876](info): set up 2 sockets
Sep 26 09:26:29 dist-test-slave-7zlw krb5kdc[14876](info): commencing operation
krb5kdc: starting...
W20260926 09:26:30.949409 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:31.080948 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:34281
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:34281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:31.180886 14889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:31.181210 14889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:31.181252 14889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:31.182731 14889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:31.182847 14889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:31.182883 14889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:31.182915 14889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:31.182945 14889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:31.184885 14889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:34281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:34281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:31.185211 14889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:31.185457 14889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:31.191243 14897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:31.191267 14894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:31.191501 14889 server_base.cc:1034] running on GCE node
W20260926 09:26:31.191257 14895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:31.191953 14889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:31.192401 14889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:31.193602 14889 hybrid_clock.cc:648] HybridClock initialized: now 1790414791193580 us; error 36 us; skew 500 ppm
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414791, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:31.196731 14889 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:31.197427 14889 webserver.cc:460] Webserver started at http://127.24.235.126:34447/ using document root <none> and password file <none>
I20260926 09:26:31.197726 14889 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:31.197804 14889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:31.197966 14889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:31.199569 14889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "0c26a4531fcf455090b291c39c3cb344"
format_stamp: "Formatted at 2026-09-26 09:26:31 on dist-test-slave-7zlw"
server_key: "63fa26c6da7777099c889a4c7bcf338c"
server_key_iv: "1a59b8df8a2541886a925d2b152c8798"
server_key_version: "encryptionkey@0"
I20260926 09:26:31.200062 14889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "0c26a4531fcf455090b291c39c3cb344"
format_stamp: "Formatted at 2026-09-26 09:26:31 on dist-test-slave-7zlw"
server_key: "63fa26c6da7777099c889a4c7bcf338c"
server_key_iv: "1a59b8df8a2541886a925d2b152c8798"
server_key_version: "encryptionkey@0"
I20260926 09:26:31.202067 14889 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 09:26:31.203029 14904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:31.203228 14889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:31.203325 14889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "0c26a4531fcf455090b291c39c3cb344"
format_stamp: "Formatted at 2026-09-26 09:26:31 on dist-test-slave-7zlw"
server_key: "63fa26c6da7777099c889a4c7bcf338c"
server_key_iv: "1a59b8df8a2541886a925d2b152c8798"
server_key_version: "encryptionkey@0"
I20260926 09:26:31.203395 14889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:31.230052 14889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:31.232705 14889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:31.232854 14889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:31.237582 14889 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:34281
I20260926 09:26:31.237622 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:34281 every 8 connection(s)
I20260926 09:26:31.238078 14889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:31.239102 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:31.241657 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344: Bootstrap starting.
I20260926 09:26:31.242350 14957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:31.242654 14957 log.cc:824] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:31.243481 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344: No bootstrap required, opened a new log
I20260926 09:26:31.245344 14957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } }
I20260926 09:26:31.245509 14957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:31.245554 14957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c26a4531fcf455090b291c39c3cb344, State: Initialized, Role: FOLLOWER
I20260926 09:26:31.245730 14957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } }
I20260926 09:26:31.245815 14957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:31.245857 14957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:31.245903 14957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:31.246809 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 14889
I20260926 09:26:31.246941 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:31.247143 25517 external_mini_cluster.cc:1428] Setting key 49d00cecf05d5d23b6a2b06651e519a6
I20260926 09:26:31.247259 14957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } }
I20260926 09:26:31.247377 14957 leader_election.cc:302] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c26a4531fcf455090b291c39c3cb344; no voters:
I20260926 09:26:31.247571 14957 leader_election.cc:288] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:31.247658 14960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:31.247849 14957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:31.247856 14960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [term 1 LEADER]: Becoming Leader. State: Replica: 0c26a4531fcf455090b291c39c3cb344, State: Running, Role: LEADER
I20260926 09:26:31.248137 14960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } }
I20260926 09:26:31.248669 14960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c26a4531fcf455090b291c39c3cb344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } } }
I20260926 09:26:31.248801 14960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:31.248986 14960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c26a4531fcf455090b291c39c3cb344. Latest consensus state: current_term: 1 leader_uuid: "0c26a4531fcf455090b291c39c3cb344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c26a4531fcf455090b291c39c3cb344" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 34281 } } }
I20260926 09:26:31.249122 14960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:31.249611 14969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:31.250929 14969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:31.252959 14969 catalog_manager.cc:1347] Generated new cluster ID: 5ea687ca1b084c6290772887ee533dc6
I20260926 09:26:31.253072 14969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:31.264621 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:31.250040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50089 (local address 127.24.235.126:34281)
0926 09:26:31.250238 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:31.250248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:31.250285 (+ 37us) server_negotiation.cc:407] Connection header received
0926 09:26:31.251951 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:31.251959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:31.252016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:31.253103 (+ 1087us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:31.253356 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:31.254786 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:31.255835 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:31.256433 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:31.256531 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:31.260452 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:31.260457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:31.260457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:31.260471 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:31.262628 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:31.263033 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:31.263037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:31.263037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:31.263159 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:31.263334 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:31.263334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:31.263334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:31.263645 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:31.263785 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:31.264212 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 09:26:31.264363 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 09:26:31.266027 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:31.267257 14969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:31.268085 14969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:31.271157 14969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c26a4531fcf455090b291c39c3cb344: Generated new TSK 0
I20260926 09:26:31.271330 14969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:31.277654 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:31.269304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34867 (local address 127.24.235.126:34281)
0926 09:26:31.269338 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:31.269340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:31.269635 (+ 295us) server_negotiation.cc:407] Connection header received
0926 09:26:31.269723 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:31.269724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:31.269744 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:31.269773 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:31.270474 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:31.271026 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:31.271962 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:31.272213 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:31.272303 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:31.273716 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:31.273717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:31.273717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:31.273725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:31.275431 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:31.276139 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:31.276140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:31.276140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:31.276217 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:31.276440 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:31.276441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:31.276442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:31.276592 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:31.276686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:31.277433 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 09:26:31.277561 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:26:31.280293 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 14889
2026-09-26T09:26:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14988](info): set up 2 sockets
Sep 26 09:26:31 dist-test-slave-7zlw krb5kdc[14988](info): commencing operation
krb5kdc: starting...
W20260926 09:26:32.325414 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:32 dist-test-slave-7zlw krb5kdc[14988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:32.456795 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43121
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:32.538718 15001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:32.538971 15001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:32.539009 15001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:32.540211 15001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:32.540284 15001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:32.540308 15001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:32.540337 15001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:32.540365 15001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:32.542004 15001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:32.542284 15001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:32.542568 15001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:32.548121 15001 server_base.cc:1034] running on GCE node
W20260926 09:26:32.548059 15009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:32.548132 15006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:32.548154 15007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:32.548524 15001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:32.548866 15001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:32.550014 15001 hybrid_clock.cc:648] HybridClock initialized: now 1790414792550011 us; error 56 us; skew 500 ppm
Sep 26 09:26:32 dist-test-slave-7zlw krb5kdc[14988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414792, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:32.552598 15001 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:32.553215 15001 webserver.cc:460] Webserver started at http://127.24.235.126:33801/ using document root <none> and password file <none>
I20260926 09:26:32.553469 15001 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:32.553539 15001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:32.553678 15001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:32.554783 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "594a5c849a154d4b8dd375770ce5b964"
format_stamp: "Formatted at 2026-09-26 09:26:32 on dist-test-slave-7zlw"
server_key: "a55650c2624b16e9f6a12e83957c7137"
server_key_iv: "c134bdba5e596fb7cf40c624fa422de9"
server_key_version: "encryptionkey@0"
I20260926 09:26:32.555146 15001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "594a5c849a154d4b8dd375770ce5b964"
format_stamp: "Formatted at 2026-09-26 09:26:32 on dist-test-slave-7zlw"
server_key: "a55650c2624b16e9f6a12e83957c7137"
server_key_iv: "c134bdba5e596fb7cf40c624fa422de9"
server_key_version: "encryptionkey@0"
I20260926 09:26:32.556514 15001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:32.557399 15016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:32.557629 15001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:32.557724 15001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "594a5c849a154d4b8dd375770ce5b964"
format_stamp: "Formatted at 2026-09-26 09:26:32 on dist-test-slave-7zlw"
server_key: "a55650c2624b16e9f6a12e83957c7137"
server_key_iv: "c134bdba5e596fb7cf40c624fa422de9"
server_key_version: "encryptionkey@0"
I20260926 09:26:32.557797 15001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:32.570967 15001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:32.573228 15001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:32.573400 15001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:32.577710 15001 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:43121
I20260926 09:26:32.577744 15068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:43121 every 8 connection(s)
I20260926 09:26:32.578119 15001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:32.579052 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:32.581405 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964: Bootstrap starting.
I20260926 09:26:32.581938 15069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:32.582302 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15001
I20260926 09:26:32.582300 15069 log.cc:824] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:32.582417 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:32.582619 25517 external_mini_cluster.cc:1428] Setting key 8f7c7ae848613cc3dc8b04a9bf565b1d
I20260926 09:26:32.583182 15069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964: No bootstrap required, opened a new log
I20260926 09:26:32.585606 15069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } }
I20260926 09:26:32.585743 15069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:32.585767 15069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594a5c849a154d4b8dd375770ce5b964, State: Initialized, Role: FOLLOWER
I20260926 09:26:32.585901 15069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } }
I20260926 09:26:32.585983 15069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:32.586014 15069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:32.586072 15069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:32.586759 15069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } }
I20260926 09:26:32.586877 15069 leader_election.cc:302] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594a5c849a154d4b8dd375770ce5b964; no voters:
I20260926 09:26:32.587088 15069 leader_election.cc:288] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:32.587133 15074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:32.587282 15074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [term 1 LEADER]: Becoming Leader. State: Replica: 594a5c849a154d4b8dd375770ce5b964, State: Running, Role: LEADER
I20260926 09:26:32.587364 15069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:32.587369 15074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } }
I20260926 09:26:32.587694 15074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594a5c849a154d4b8dd375770ce5b964. Latest consensus state: current_term: 1 leader_uuid: "594a5c849a154d4b8dd375770ce5b964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } } }
I20260926 09:26:32.587761 15074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:32.588011 15075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594a5c849a154d4b8dd375770ce5b964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594a5c849a154d4b8dd375770ce5b964" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 43121 } } }
I20260926 09:26:32.588112 15075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:32.588795 15078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:32.589345 15078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:32.590173 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:32.583157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52831 (local address 127.24.235.126:43121)
0926 09:26:32.583358 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:32.583367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:32.583400 (+ 33us) server_negotiation.cc:407] Connection header received
0926 09:26:32.584344 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:32.584351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:32.584399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:32.584553 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:32.584761 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:32.585695 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:32.586877 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:32.587922 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:32.588035 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:32.589188 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:32.589192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:32.589193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:32.589416 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:32.589466 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:32.589481 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:32.589844 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:32.589877 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 09:26:32.589946 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260926 09:26:32.591102 15078 catalog_manager.cc:1347] Generated new cluster ID: f6046783e3874f048bce60a15007f9c1
I20260926 09:26:32.591153 15078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:32.591282 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:32.596333 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:32.593473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55995 (local address 127.24.235.126:43121)
0926 09:26:32.593512 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:32.593514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:32.593643 (+ 129us) server_negotiation.cc:407] Connection header received
0926 09:26:32.593709 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:32.593710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:32.593721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:32.593772 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:32.593927 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:32.594403 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:32.595124 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:32.595338 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:32.595417 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:32.596086 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:32.596087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:32.596088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:32.596092 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:32.596106 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:32.596110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:32.596161 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:32.596230 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 09:26:32.596253 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 09:26:32.612026 15078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:32.612735 15078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:32.617415 15078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594a5c849a154d4b8dd375770ce5b964: Generated new TSK 0
I20260926 09:26:32.617637 15078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:32.623116 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15001
2026-09-26T09:26:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:32 dist-test-slave-7zlw krb5kdc[15100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:32 dist-test-slave-7zlw krb5kdc[15100](info): set up 2 sockets
Sep 26 09:26:32 dist-test-slave-7zlw krb5kdc[15100](info): commencing operation
krb5kdc: starting...
W20260926 09:26:33.665272 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:33 dist-test-slave-7zlw krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:33.792994 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36069
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:33.874886 15113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:33.875114 15113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:33.875137 15113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:33.876248 15113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:33.876286 15113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:33.876299 15113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:33.876312 15113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:33.876324 15113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:33.877790 15113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:33.877985 15113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:33.878157 15113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:33.883332 15113 server_base.cc:1034] running on GCE node
W20260926 09:26:33.883273 15119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:33.883311 15118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:33.883280 15121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:33.883782 15113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:33.884121 15113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:33.885326 15113 hybrid_clock.cc:648] HybridClock initialized: now 1790414793885347 us; error 110 us; skew 500 ppm
Sep 26 09:26:33 dist-test-slave-7zlw krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414793, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:33.893111 15113 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:33.893672 15113 webserver.cc:460] Webserver started at http://127.24.235.126:37951/ using document root <none> and password file <none>
I20260926 09:26:33.893918 15113 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:33.893988 15113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:33.894138 15113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:33.895377 15113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "eacad8eab31044f1a60325fe1c3f27a8"
format_stamp: "Formatted at 2026-09-26 09:26:33 on dist-test-slave-7zlw"
server_key: "1fa2c85936e2d3c9072812ef43052c73"
server_key_iv: "e2cdacb01a86541d960a5da4c2a6e48b"
server_key_version: "encryptionkey@0"
I20260926 09:26:33.895774 15113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "eacad8eab31044f1a60325fe1c3f27a8"
format_stamp: "Formatted at 2026-09-26 09:26:33 on dist-test-slave-7zlw"
server_key: "1fa2c85936e2d3c9072812ef43052c73"
server_key_iv: "e2cdacb01a86541d960a5da4c2a6e48b"
server_key_version: "encryptionkey@0"
I20260926 09:26:33.897411 15113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:33.898268 15128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:33.898475 15113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:33.898638 15113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "eacad8eab31044f1a60325fe1c3f27a8"
format_stamp: "Formatted at 2026-09-26 09:26:33 on dist-test-slave-7zlw"
server_key: "1fa2c85936e2d3c9072812ef43052c73"
server_key_iv: "e2cdacb01a86541d960a5da4c2a6e48b"
server_key_version: "encryptionkey@0"
I20260926 09:26:33.898725 15113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:33.902496 15113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:33.905006 15113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:33.905243 15113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:33.909747 15113 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36069
I20260926 09:26:33.909781 15180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36069 every 8 connection(s)
I20260926 09:26:33.910133 15113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:33.911087 15181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:33.913550 15181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8: Bootstrap starting.
I20260926 09:26:33.914181 15181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:33.914494 15181 log.cc:824] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:33.915230 15181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8: No bootstrap required, opened a new log
I20260926 09:26:33.916862 15181 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } }
I20260926 09:26:33.917059 15181 raft_consensus.cc:374] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 09:26:33.917137 15181 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacad8eab31044f1a60325fe1c3f27a8, State: Initialized, Role: FOLLOWER
I20260926 09:26:33.917311 15181 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } }
I20260926 09:26:33.917397 15181 raft_consensus.cc:388] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 09:26:33.917443 15181 raft_consensus.cc:482] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 09:26:33.917491 15181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:33.918383 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15113
I20260926 09:26:33.918506 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:33.918592 15181 raft_consensus.cc:504] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } }
I20260926 09:26:33.918725 25517 external_mini_cluster.cc:1428] Setting key 3588e2731cc8f9e32d0238c5692f0659
I20260926 09:26:33.918706 15181 leader_election.cc:302] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eacad8eab31044f1a60325fe1c3f27a8; no voters:
I20260926 09:26:33.918895 15181 leader_election.cc:288] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 09:26:33.918957 15184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:33.919123 15184 raft_consensus.cc:686] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [term 1 LEADER]: Becoming Leader. State: Replica: eacad8eab31044f1a60325fe1c3f27a8, State: Running, Role: LEADER
I20260926 09:26:33.919199 15181 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:33.919230 15184 consensus_queue.cc:237] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } }
I20260926 09:26:33.919575 15184 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eacad8eab31044f1a60325fe1c3f27a8. Latest consensus state: current_term: 1 leader_uuid: "eacad8eab31044f1a60325fe1c3f27a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } } }
I20260926 09:26:33.919749 15184 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:33.919586 15185 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eacad8eab31044f1a60325fe1c3f27a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacad8eab31044f1a60325fe1c3f27a8" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36069 } } }
I20260926 09:26:33.919879 15185 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:33.920049 15193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:33.920993 15193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:33.921871 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:33.919481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52241 (local address 127.24.235.126:36069)
0926 09:26:33.920005 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:33.920022 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:33.920065 (+ 43us) server_negotiation.cc:407] Connection header received
0926 09:26:33.920182 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:33.920189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:33.920224 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:33.920430 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:33.921324 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:33.921327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:33.921327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:33.921332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:33.921378 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:33.921394 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:33.921472 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:33.921634 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 09:26:33.921679 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20260926 09:26:33.922628 15193 catalog_manager.cc:1347] Generated new cluster ID: 14c9ee91afe44702ac4f14f77516f4d7
I20260926 09:26:33.922675 15193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:33.925340 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:33.929623 15193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:33.929802 15193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:33.930681 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:33.926923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54153 (local address 127.24.235.126:36069)
0926 09:26:33.926958 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:33.926960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:33.929604 (+ 2644us) server_negotiation.cc:407] Connection header received
0926 09:26:33.929712 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:33.929713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:33.929719 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:33.929781 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:33.930459 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:33.930460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:33.930460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 09:26:33.930465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:33.930481 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 09:26:33.930485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:33.930539 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:33.930602 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 09:26:33.930617 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 09:26:33.938401 15193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eacad8eab31044f1a60325fe1c3f27a8: Generated new TSK 0
I20260926 09:26:33.938613 15193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 09:26:33.941164 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15113
2026-09-26T09:26:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1318 ms)
[----------] 18 tests from AuthTokenIssuingTest (24333 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:34 dist-test-slave-7zlw krb5kdc[15212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:34 dist-test-slave-7zlw krb5kdc[15212](info): set up 2 sockets
Sep 26 09:26:34 dist-test-slave-7zlw krb5kdc[15212](info): commencing operation
krb5kdc: starting...
W20260926 09:26:34.989326 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:35.125325 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45339
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:35.208320 15225 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:35.208549 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:35.208567 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:35.208590 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:35.209758 15225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:35.209796 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:35.209810 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:35.209822 15225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:35.209834 15225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:35.211290 15225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:35.211495 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:35.211678 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:35.216887 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:35.216988 15233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:35.216961 15225 server_base.cc:1034] running on GCE node
W20260926 09:26:35.216998 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:35.217495 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:35.217772 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:35.218938 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790414795218909 us; error 46 us; skew 500 ppm
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:35.221807 15225 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:35.222313 15225 webserver.cc:460] Webserver started at http://127.24.235.126:43431/ using document root <none> and password file <none>
I20260926 09:26:35.222545 15225 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:35.222605 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:35.222774 15225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:35.223903 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "edf93d0cdbf3218e9be62069116ff176"
server_key_iv: "e73ae63fb12bc2dfbfec55160404cb64"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.224339 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "edf93d0cdbf3218e9be62069116ff176"
server_key_iv: "e73ae63fb12bc2dfbfec55160404cb64"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.225906 15225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 09:26:35.226631 15240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:35.226917 15225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 09:26:35.227013 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "edf93d0cdbf3218e9be62069116ff176"
server_key_iv: "e73ae63fb12bc2dfbfec55160404cb64"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.227094 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:35.254694 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:35.257251 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:35.257386 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:35.261927 15225 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:45339
I20260926 09:26:35.261960 15292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:45339 every 8 connection(s)
I20260926 09:26:35.262357 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:35.263381 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:35.264963 15293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:35.271416 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15225
I20260926 09:26:35.271518 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:35.271714 25517 external_mini_cluster.cc:1428] Setting key c7d31726f1d90ba4b1cc0a433b45db5c
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:35.276577 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.265792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45339 (local address 127.0.0.1:46174)
0926 09:26:35.266057 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.266105 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.266722 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.267032 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.267035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.267328 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.267484 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.267488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.268591 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.268592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.269351 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.269354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.269876 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.269877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.269943 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.270570 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.270580 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.272367 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.274457 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.274464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.274467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.274854 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.275158 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.275159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.275159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.275285 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.275817 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.275825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.275917 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.276045 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.276263 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":131,"threads_started":1}
I20260926 09:26:35.276589 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.265897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.24.235.126:45339)
0926 09:26:35.266068 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.266077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.266218 (+ 141us) server_negotiation.cc:407] Connection header received
0926 09:26:35.266897 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.266899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.266925 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.267026 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.267625 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.268470 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.269464 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.269776 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.269903 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.272467 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.272477 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.272477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.272487 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.274364 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.274968 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.274971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.274971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.275072 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.275381 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.275382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.275383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.275720 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.275833 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.276133 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.276389 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260926 09:26:35.277563 15293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:35.277984 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.277828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45666)
0926 09:26:35.277886 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.277947 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:26:35.278147 15244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
I20260926 09:26:35.278709 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.278579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45682)
0926 09:26:35.278640 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.278677 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 09:26:35.278841 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:35443: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
I20260926 09:26:35.301358 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:35443
--webserver_interface=127.24.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:26:35.328114 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } attempt: 1
I20260926 09:26:35.328790 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.328634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45694)
0926 09:26:35.328703 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.328754 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:26:35.329310 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.329177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45700)
0926 09:26:35.329254 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.329286 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 09:26:35.329489 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:35443: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
W20260926 09:26:35.385183 15300 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:35.385462 15300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:35.385493 15300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:35.385540 15300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:35.386724 15300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:35.386803 15300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:35.386827 15300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:35.386848 15300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:35.386878 15300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:35.388427 15300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:35443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:35.388742 15300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:35.388988 15300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:35.394586 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:35.394536 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:35.394769 15300 server_base.cc:1034] running on GCE node
W20260926 09:26:35.394536 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:35.395097 15300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:35.395449 15300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:35.396663 15300 hybrid_clock.cc:648] HybridClock initialized: now 1790414795396646 us; error 51 us; skew 500 ppm
I20260926 09:26:35.397655 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } attempt: 2
I20260926 09:26:35.398102 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.397951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45712)
0926 09:26:35.398028 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.398070 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 09:26:35.398489 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.398383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45720)
0926 09:26:35.398432 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.398460 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:26:35.398594 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:35443: Network error: Client connection negotiation failed: client connection to 127.24.235.125:35443: connect: Connection refused (error 111)
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:35.399663 15300 init.cc:377] Logged in from keytab as kudu/127.24.235.125@KRBTEST.COM (short username kudu)
I20260926 09:26:35.400254 15300 webserver.cc:460] Webserver started at http://127.24.235.125:33945/ using document root <none> and password file <none>
I20260926 09:26:35.400542 15300 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:35.400641 15300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:35.400839 15300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:35.402189 15300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data/instance:
uuid: "1f773df2e6264f0b901886ed3e7e44be"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "f01a00588b38e83d5b73d93b88ad10ca"
server_key_iv: "cfa4797a224fd613f8f4e9e7d92a2a1c"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.402633 15300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal/instance:
uuid: "1f773df2e6264f0b901886ed3e7e44be"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "f01a00588b38e83d5b73d93b88ad10ca"
server_key_iv: "cfa4797a224fd613f8f4e9e7d92a2a1c"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.404240 15300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:35.405133 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:35.405321 15300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:35.405426 15300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "1f773df2e6264f0b901886ed3e7e44be"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "f01a00588b38e83d5b73d93b88ad10ca"
server_key_iv: "cfa4797a224fd613f8f4e9e7d92a2a1c"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.405519 15300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:35.437191 15300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:35.439781 15300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:35.439965 15300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:35.444727 15300 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.125:35443
I20260926 09:26:35.444777 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.125:35443 every 8 connection(s)
I20260926 09:26:35.445386 15300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
I20260926 09:26:35.446332 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:35.447255 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15300
I20260926 09:26:35.447359 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-1/wal/instance
I20260926 09:26:35.447572 25517 external_mini_cluster.cc:1428] Setting key da302a72a112c2177159f311a2873ae0
I20260926 09:26:35.448740 15368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.235.124@KRBTEST.COM; defaulting to no policy
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.124@KRBTEST.COM" created.
I20260926 09:26:35.459537 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.449765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.24.235.126:45339)
0926 09:26:35.449828 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.449831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.450101 (+ 270us) server_negotiation.cc:407] Connection header received
0926 09:26:35.450650 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.450652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.450669 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.450725 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.451486 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.452433 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.453644 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.453987 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.454067 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.456375 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.456376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.456377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.456383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.457825 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.458327 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.458328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.458329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.458381 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.458800 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.458801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.458801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.458947 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.459014 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.459346 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.459438 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 09:26:35.459748 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.449766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45339 (local address 127.0.0.1:46182)
0926 09:26:35.449949 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.449988 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.450515 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.450808 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.450812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.451101 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.451330 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.451335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.452648 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.452649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.453540 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.453542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.454089 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.454090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.454199 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.454813 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.454825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.456280 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.457902 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.457909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.457911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.458256 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.458483 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.458484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.458484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.458633 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.459046 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.459052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.459190 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.459320 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.459490 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 09:26:35.460508 15368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:35.468683 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.460801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45734)
0926 09:26:35.460853 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.460867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.460914 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.461204 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.461205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.461376 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.461444 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.461446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.462326 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.462327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.463199 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.463203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.463667 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.463675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.463755 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.464148 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.464154 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.465612 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.467674 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.467674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.467675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.467843 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.468066 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.468066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.468067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.468117 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.468437 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.468437 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.468477 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.468533 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.468613 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:26:35.468842 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.460873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.24.235.125:35443)
0926 09:26:35.461051 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.461055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.461075 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:26:35.461105 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.461107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.461119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.461183 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.461523 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.462237 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.463286 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.463586 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.463644 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.465704 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.465714 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.465714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.465722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.467585 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.467932 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.467936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.467936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.467997 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.468183 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.468184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.468185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.468365 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.468433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.468577 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.468663 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260926 09:26:35.469551 15368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:35.469938 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.469796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54942)
0926 09:26:35.469872 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.469917 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 09:26:35.470103 15317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
I20260926 09:26:35.470579 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.470473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54946)
0926 09:26:35.470521 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.470551 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:26:35.470709 15368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
I20260926 09:26:35.476125 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:43201
--webserver_interface=127.24.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:26:35.489756 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } attempt: 3
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:35.497828 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.490118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45744)
0926 09:26:35.490211 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.490222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.490264 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.490442 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.490443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.490592 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.490674 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.490675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.491642 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.491642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.492177 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.492179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.492728 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.492729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.492799 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.493359 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.493364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.494791 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.496656 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.496657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.496658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.496830 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.497128 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.497129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.497130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.497208 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.497531 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.497533 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.497593 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.497669 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.497754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 09:26:35.497887 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.490201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.24.235.125:35443)
0926 09:26:35.490281 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.490283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.490297 (+ 14us) server_negotiation.cc:407] Connection header received
0926 09:26:35.490331 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.490333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.490348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.490428 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.490757 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.491561 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.492262 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.492618 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.492687 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.494883 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.494885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.494885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.494895 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.496558 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.496909 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.496911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.496911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.496975 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.497287 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.497288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.497288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.497429 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.497499 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.497711 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.497797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 09:26:35.498353 15293 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:35.498764 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.498653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54958)
0926 09:26:35.498705 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.498735 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:26:35.499126 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.499045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54966)
0926 09:26:35.499088 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.499111 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:26:35.499235 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
I20260926 09:26:35.519948 15368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 1
I20260926 09:26:35.520571 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.520401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54982)
0926 09:26:35.520478 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.520531 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:26:35.521080 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.520957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:54996)
0926 09:26:35.521004 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.521059 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:26:35.521181 15368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
I20260926 09:26:35.530364 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 1
I20260926 09:26:35.530812 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.530684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55010)
0926 09:26:35.530743 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.530782 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 09:26:35.531177 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.531077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55014)
0926 09:26:35.531128 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.531159 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 09:26:35.531265 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
W20260926 09:26:35.562637 15375 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:35.562861 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:35.562879 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:35.562899 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:35.564070 15375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:35.564112 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:35.564126 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:35.564139 15375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:35.564152 15375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:35.565675 15375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:45339,127.24.235.125:35443,127.24.235.124:43201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:35.565938 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:35.566183 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:35.571434 15380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:35.571450 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:35.571465 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:35.571771 15375 server_base.cc:1034] running on GCE node
I20260926 09:26:35.572017 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:35.572309 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:35.573451 15375 hybrid_clock.cc:648] HybridClock initialized: now 1790414795573437 us; error 27 us; skew 500 ppm
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:35.575943 15375 init.cc:377] Logged in from keytab as kudu/127.24.235.124@KRBTEST.COM (short username kudu)
I20260926 09:26:35.576404 15375 webserver.cc:460] Webserver started at http://127.24.235.124:45045/ using document root <none> and password file <none>
I20260926 09:26:35.576651 15375 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:35.576720 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:35.576889 15375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:35.578061 15375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data/instance:
uuid: "f8367ebc32274a0e8a720bb4a3d2d558"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "b004563e0d3adf4c4347e8a3a408c466"
server_key_iv: "492230353d363bf7f900603d6dc7f6a9"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.578372 15375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal/instance:
uuid: "f8367ebc32274a0e8a720bb4a3d2d558"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "b004563e0d3adf4c4347e8a3a408c466"
server_key_iv: "492230353d363bf7f900603d6dc7f6a9"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.579669 15375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:35.580463 15390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:35.580669 15375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:35.580770 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "f8367ebc32274a0e8a720bb4a3d2d558"
format_stamp: "Formatted at 2026-09-26 09:26:35 on dist-test-slave-7zlw"
server_key: "b004563e0d3adf4c4347e8a3a408c466"
server_key_iv: "492230353d363bf7f900603d6dc7f6a9"
server_key_version: "encryptionkey@0"
I20260926 09:26:35.580825 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:35.589303 15368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 2
I20260926 09:26:35.589748 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.589589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55022)
0926 09:26:35.589663 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.589709 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:26:35.590102 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.590022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55038)
0926 09:26:35.590062 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.590086 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 09:26:35.590191 15368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
I20260926 09:26:35.605475 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:35.606403 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 2
I20260926 09:26:35.606807 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.606684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55048)
0926 09:26:35.606744 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.606781 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:26:35.607221 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.607064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55050)
0926 09:26:35.607137 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.607184 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 09:26:35.607406 15293 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43201: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43201: connect: Connection refused (error 111)
I20260926 09:26:35.608397 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:35.608566 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:35.613485 15375 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.124:43201
I20260926 09:26:35.613513 15442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.124:43201 every 8 connection(s)
I20260926 09:26:35.613974 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
I20260926 09:26:35.614923 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:35.616478 15443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:35.621533 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15375
I20260926 09:26:35.621655 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790414522268193-25517-0/minicluster-data/master-2/wal/instance
I20260926 09:26:35.621888 25517 external_mini_cluster.cc:1428] Setting key 9a2e7c142710f566696dc2898e22ee4c
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:35.626024 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.617222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.24.235.126:45339)
0926 09:26:35.617266 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.617270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.617497 (+ 227us) server_negotiation.cc:407] Connection header received
0926 09:26:35.617933 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.617934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.617943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.618001 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.618586 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.619077 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.620109 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.620335 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.620411 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.623161 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.623163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.623163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.623171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.624589 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.625040 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.625041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.625042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.625106 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.625381 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.625383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.625383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.625507 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.625591 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.625855 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.625935 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:35.626223 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.617201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:45339 (local address 127.0.0.1:46190)
0926 09:26:35.617369 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.617411 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.617854 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.618050 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.618053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.618290 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.618479 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.618483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.619174 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.619174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.619998 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.620000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.620435 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.620436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.620544 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.621181 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.621189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.623072 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.624690 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.624697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.624699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.624943 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.625197 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.625197 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.625198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.625310 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.625581 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.625588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.625725 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.625833 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.625977 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260926 09:26:35.626830 15443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:35.630669 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.622311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.24.235.124:43201)
0926 09:26:35.622489 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.622494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.622523 (+ 29us) server_negotiation.cc:407] Connection header received
0926 09:26:35.622612 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.622615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.622638 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.622719 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.623327 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.623984 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.624746 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.625105 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.625184 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.627688 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.627696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.627696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.627703 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.629334 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.629782 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.629785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.629786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.629847 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.630062 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.630063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.630063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.630243 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.630359 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.630484 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.630542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:35.634609 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.627121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:35443 (local address 127.0.0.1:45748)
0926 09:26:35.627202 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.627214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.627264 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.627378 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.627379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.627524 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.627589 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.627590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.628509 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.628510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.629154 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.629157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.629704 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.629705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.629824 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.630219 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.630223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.631809 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.633512 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.633512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.633513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.633726 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.633940 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.633941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.633941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.634005 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.634305 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.634306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.634340 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.634394 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.634482 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 09:26:35.634676 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.627179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.24.235.125:35443)
0926 09:26:35.627223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.627225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.627250 (+ 25us) server_negotiation.cc:407] Connection header received
0926 09:26:35.627321 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.627321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.627329 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.627373 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.627698 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.628425 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.629234 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.629545 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.629637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.631887 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.631889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.631889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.631896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.633440 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.633803 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.633804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.633805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.633867 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.634074 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.634075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.634075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.634235 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.634307 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.634471 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.634551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:35.635197 15443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:35.639235 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.631483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.24.235.126:45339)
0926 09:26:35.631528 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.631530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.631561 (+ 31us) server_negotiation.cc:407] Connection header received
0926 09:26:35.631617 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.631618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.631628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.631691 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.632206 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.632648 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.633371 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.633625 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.633698 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.636237 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.636238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.636238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.636245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.637913 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.638321 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.638322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.638322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.638378 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.638684 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.638685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.638685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.638796 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.639027 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.639063 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.639134 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:26:35.642328 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.635475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55062)
0926 09:26:35.635502 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.635512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.635551 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.635677 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.635678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.635805 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.635827 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.635829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.636686 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.636688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.637400 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.637403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.637894 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.637894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.637934 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.638283 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.638288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.639557 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.641322 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.641323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.641323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.641483 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.641690 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.641690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.641691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.641741 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.641988 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.641989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.642024 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.642107 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.642205 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260926 09:26:35.642366 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.635504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.24.235.124:43201)
0926 09:26:35.635534 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.635536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.635543 (+ 7us) server_negotiation.cc:407] Connection header received
0926 09:26:35.635613 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.635614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.635623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.635674 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.635918 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.636615 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.637479 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.637811 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.637898 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.639683 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.639684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.639684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.639690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.641231 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.641563 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.641563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.641564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.641611 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.641808 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.641809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.641809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.641913 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.641983 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.642129 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.642252 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":17}
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:35.643898 15443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: Bootstrap starting.
I20260926 09:26:35.644888 15443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:35.645234 15443 log.cc:824] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:35.646021 15443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: No bootstrap required, opened a new log
I20260926 09:26:35.647082 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.639952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.24.235.125:35443)
0926 09:26:35.640002 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.640004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.640206 (+ 202us) server_negotiation.cc:407] Connection header received
0926 09:26:35.640234 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.640235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.640247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.640313 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.640740 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.641264 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.641901 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.642094 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.642181 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.644303 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.644305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.644305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.644312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.645925 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.646319 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.646320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.646320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.646380 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.646581 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.646582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.646582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.646691 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.646885 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.646923 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.646991 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 09:26:35.647765 15443 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.647907 15443 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8367ebc32274a0e8a720bb4a3d2d558, State: Initialized, Role: FOLLOWER
I20260926 09:26:35.648011 15443 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.648331 15443 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:35.648350 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.648466 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.648679 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 09:26:35.650271 15460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:35.650328 15460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:35.650074 15267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46194
I20260926 09:26:35.650530 15375 master_runner.cc:186] Error getting master registration for 127.24.235.126:45339: OK, instance_id { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" instance_seqno: 1790414795260544 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:26:35.652966 15375 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:26:35.681331 15368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 3
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:35.688753 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.681804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.24.235.124:43201)
0926 09:26:35.681881 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.681883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.681893 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:35.681937 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.681938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.681949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.682026 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.682370 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.682900 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.683675 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.683870 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.683949 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.685880 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.685881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.685881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.685889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.687476 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.687811 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.687812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.687813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.687869 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.688137 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.688138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.688138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.688326 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.688392 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.688567 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.688649 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:26:35.688812 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.681731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55068)
0926 09:26:35.681799 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.681812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.681862 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.682043 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.682044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.682230 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.682279 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.682281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.682995 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.682996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.683586 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.683588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.683957 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.683958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.684018 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.684530 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.684535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.685791 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.687558 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.687559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.687560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.687735 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.687965 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.687967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.687967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.688054 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.688405 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.688406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.688456 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.688533 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.688701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 09:26:35.690986 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: Bootstrap starting.
I20260926 09:26:35.691749 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:35.692051 15368 log.cc:824] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:35.692974 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: No bootstrap required, opened a new log
I20260926 09:26:35.695247 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.695406 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f773df2e6264f0b901886ed3e7e44be, State: Initialized, Role: FOLLOWER
I20260926 09:26:35.695498 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.695786 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.695850 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:35.695916 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:26:35.698040 15474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:35.698154 15474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:35.698305 15300 master_runner.cc:186] Error getting master registration for 127.24.235.126:45339: OK, instance_id { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" instance_seqno: 1790414795260544 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:26:35.701189 15300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:26:35.706578 15293 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } attempt: 3
Sep 26 09:26:35 dist-test-slave-7zlw krb5kdc[15212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414795, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:35.714236 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.706940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.24.235.124:43201)
0926 09:26:35.706982 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.706984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.706993 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:35.707085 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.707086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.707096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.707153 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.707465 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.708197 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.708992 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.709307 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.709378 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.711592 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.711594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.711595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.711602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.713078 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.713415 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.713416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.713417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.713498 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.713709 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.713710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.713710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.713840 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.713896 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.714071 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.714149 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 09:26:35.714236 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.706892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43201 (local address 127.0.0.1:55074)
0926 09:26:35.706945 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:35.706957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:35.707001 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:35.707162 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:35.707164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:35.707338 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:35.707379 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.707381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.708280 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.708281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.708904 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:35.708906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.709388 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.709389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:35.709447 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.709949 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:35.709952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:35.711496 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:35.713159 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.713160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.713160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.713340 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.713555 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:35.713556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:35.713556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:35.713638 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:35.713919 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:35.713920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:35.713965 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 09:26:35.714032 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 09:26:35.714131 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 09:26:35.716233 15293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: Bootstrap starting.
I20260926 09:26:35.716977 15293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:35.717315 15293 log.cc:824] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:35.718154 15293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: No bootstrap required, opened a new log
I20260926 09:26:35.719532 15293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.719676 15293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170f657ec9bc4ecc8ac3d6595ae0ce52, State: Initialized, Role: FOLLOWER
I20260926 09:26:35.719761 15293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.720098 15293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:35.720126 15267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:26:35.720170 15267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:35.721099 15267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.721395 15267 leader_election.cc:288] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [CANDIDATE]: Term 1 election: Requested vote from peers 1f773df2e6264f0b901886ed3e7e44be (127.24.235.125:35443), f8367ebc32274a0e8a720bb4a3d2d558 (127.24.235.124:43201)
I20260926 09:26:35.722003 15342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f773df2e6264f0b901886ed3e7e44be"
I20260926 09:26:35.722172 15342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [term 0 FOLLOWER]: Advancing to term 1
W20260926 09:26:35.722332 15488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:35.722374 15488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:35.723145 15342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170f657ec9bc4ecc8ac3d6595ae0ce52 in term 1.
I20260926 09:26:35.723677 15417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8367ebc32274a0e8a720bb4a3d2d558"
I20260926 09:26:35.723796 15417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:35.724587 15417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170f657ec9bc4ecc8ac3d6595ae0ce52 in term 1.
I20260926 09:26:35.726351 15242 leader_election.cc:302] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170f657ec9bc4ecc8ac3d6595ae0ce52, f8367ebc32274a0e8a720bb4a3d2d558; no voters:
I20260926 09:26:35.728328 15225 master_runner.cc:263] No leader master found from master 170f657ec9bc4ecc8ac3d6595ae0ce52
I20260926 09:26:35.728374 15225 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:26:35.720064 15477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.728596 15477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.728705 15477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:35.728794 15477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [term 1 LEADER]: Becoming Leader. State: Replica: 170f657ec9bc4ecc8ac3d6595ae0ce52, State: Running, Role: LEADER
I20260926 09:26:35.728927 15477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } }
I20260926 09:26:35.729512 15477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170f657ec9bc4ecc8ac3d6595ae0ce52. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.729660 15477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:35.730836 15496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:35.731094 15496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:35.740962 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.730015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.24.235.124:43201)
0926 09:26:35.730058 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.730060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.733455 (+ 3395us) server_negotiation.cc:407] Connection header received
0926 09:26:35.733546 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.733548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.733559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.733592 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.734238 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.734986 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.736063 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.736360 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.736412 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.738066 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.738067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.738067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.738074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.739714 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.740106 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.740107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.740108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.740161 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.740368 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.740368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.740369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.740490 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.740560 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.740787 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.740861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:35.743041 15417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [term 1 FOLLOWER]: Refusing update from remote peer 170f657ec9bc4ecc8ac3d6595ae0ce52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:35.743417 15494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:35.744870 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170f657ec9bc4ecc8ac3d6595ae0ce52. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.744984 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.745246 15342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [term 1 FOLLOWER]: Refusing update from remote peer 170f657ec9bc4ecc8ac3d6595ae0ce52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:35.745554 15494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:35.746767 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170f657ec9bc4ecc8ac3d6595ae0ce52. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.746865 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.747982 15494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.748082 15494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:35.748551 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.748636 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.748890 15500 mvcc.cc:205] Tried to move back new op lower bound from 7333539003360821248 to 7333539003306921984. Current Snapshot: MvccSnapshot[applied={T|T < 7333539003360821248}]
I20260926 09:26:35.749447 15503 mvcc.cc:205] Tried to move back new op lower bound from 7333539003360821248 to 7333539003306921984. Current Snapshot: MvccSnapshot[applied={T|T < 7333539003360821248}]
I20260926 09:26:35.750048 15496 catalog_manager.cc:1347] Generated new cluster ID: 371548742fc045a098aaa2634d99a6e2
I20260926 09:26:35.750092 15496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:35.751255 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.730300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.24.235.126:45339)
0926 09:26:35.730343 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.730345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.733287 (+ 2942us) server_negotiation.cc:407] Connection header received
0926 09:26:35.733369 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.733370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.733383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.733439 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.744745 (+ 11306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.745418 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.746344 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.746602 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.746645 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.748161 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.748163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.748163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.748170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.750163 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.750522 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.750523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.750523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.750571 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.750742 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.750743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.750743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.750850 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.750921 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.751091 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.751172 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 09:26:35.751259 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:35.730686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.24.235.125:35443)
0926 09:26:35.730729 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.730731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.742023 (+ 11292us) server_negotiation.cc:407] Connection header received
0926 09:26:35.742078 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.742079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.742093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.742169 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.742816 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.743573 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.745215 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:35.745532 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:35.745586 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:35.747314 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:35.747315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:35.747315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:35.747324 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:35.749698 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.750243 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.750244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.750245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.750341 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:35.750596 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:35.750597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:35.750597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:35.750738 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:35.750818 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:35.751079 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 09:26:35.751166 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:35.756911 15477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.757700 15449 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170f657ec9bc4ecc8ac3d6595ae0ce52" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 45339 } } peers { permanent_uuid: "1f773df2e6264f0b901886ed3e7e44be" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 35443 } } peers { permanent_uuid: "f8367ebc32274a0e8a720bb4a3d2d558" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43201 } } }
I20260926 09:26:35.757786 15449 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:35.757910 15501 mvcc.cc:205] Tried to move back new op lower bound from 7333539003360821248 to 7333539003306921984. Current Snapshot: MvccSnapshot[applied={T|T < 7333539003360821248}]
I20260926 09:26:35.757006 15477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:35.770912 15496 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:35.771755 15496 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:35.776921 15496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 170f657ec9bc4ecc8ac3d6595ae0ce52: Generated new TSK 0
I20260926 09:26:35.777187 15496 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:26:35.805855 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.804085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.24.235.126:45339)
0926 09:26:35.804131 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.804133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.805161 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 09:26:35.805270 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.805271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.805282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.805312 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.805782 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46204: BlockingRecv error: recv got EOF from 127.0.0.1:46204 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:26:35.806842 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.804443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.24.235.125:35443)
0926 09:26:35.804488 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.804490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.806228 (+ 1738us) server_negotiation.cc:407] Connection header received
0926 09:26:35.806391 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.806392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.806403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.806432 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.806783 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45780: BlockingRecv error: recv got EOF from 127.0.0.1:45780 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 09:26:35.807647 15447 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.804881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.24.235.124:43201)
0926 09:26:35.804918 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.804919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.807140 (+ 2221us) server_negotiation.cc:407] Connection header received
0926 09:26:35.807236 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.807237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.807247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.807279 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.807563 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55084: BlockingRecv error: recv got EOF from 127.0.0.1:55084 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 09:26:35.810181 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.809453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.24.235.125:35443)
0926 09:26:35.809486 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.809488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.809698 (+ 210us) server_negotiation.cc:407] Connection header received
0926 09:26:35.809801 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.809802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.809819 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.809847 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.810138 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45784: BlockingRecv error: recv got EOF from 127.0.0.1:45784 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:26:35.813184 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.812403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.24.235.125:35443)
0926 09:26:35.812433 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.812435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.812683 (+ 248us) server_negotiation.cc:407] Connection header received
0926 09:26:35.812775 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.812776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.812786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.812813 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.813141 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45796: BlockingRecv error: recv got EOF from 127.0.0.1:45796 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:26:35.817128 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.816360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.24.235.125:35443)
0926 09:26:35.816392 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.816394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.816628 (+ 234us) server_negotiation.cc:407] Connection header received
0926 09:26:35.816721 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.816722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.816732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.816759 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.817087 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45810: BlockingRecv error: recv got EOF from 127.0.0.1:45810 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:26:35.822976 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.822221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.24.235.125:35443)
0926 09:26:35.822254 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.822255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.822486 (+ 231us) server_negotiation.cc:407] Connection header received
0926 09:26:35.822577 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.822578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.822588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.822615 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.822936 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45820: BlockingRecv error: recv got EOF from 127.0.0.1:45820 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 09:26:35.833302 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.832332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.24.235.125:35443)
0926 09:26:35.832377 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.832379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.832673 (+ 294us) server_negotiation.cc:407] Connection header received
0926 09:26:35.832783 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.832784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.832797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.832827 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.833242 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45822: BlockingRecv error: recv got EOF from 127.0.0.1:45822 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 09:26:35.851768 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.850972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.24.235.125:35443)
0926 09:26:35.851026 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.851028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.851139 (+ 111us) server_negotiation.cc:407] Connection header received
0926 09:26:35.851185 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.851186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.851200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.851292 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.851699 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45832: BlockingRecv error: recv got EOF from 127.0.0.1:45832 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 09:26:35.885814 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.884953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.24.235.125:35443)
0926 09:26:35.885003 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.885005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.885190 (+ 185us) server_negotiation.cc:407] Connection header received
0926 09:26:35.885233 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.885234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.885250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.885323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.885749 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45846: BlockingRecv error: recv got EOF from 127.0.0.1:45846 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 09:26:35.951586 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:35.950904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.24.235.125:35443)
0926 09:26:35.950998 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:35.951000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:35.951064 (+ 64us) server_negotiation.cc:407] Connection header received
0926 09:26:35.951096 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:35.951097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:35.951108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:35.951166 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:35.951529 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45856: BlockingRecv error: recv got EOF from 127.0.0.1:45856 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 09:26:36.081737 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:36.080887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.24.235.125:35443)
0926 09:26:36.080980 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.080983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.081108 (+ 125us) server_negotiation.cc:407] Connection header received
0926 09:26:36.081216 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.081217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.081229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.081310 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:36.081666 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45872: BlockingRecv error: recv got EOF from 127.0.0.1:45872 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 09:26:36.339725 15373 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:36.339091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.24.235.125:35443)
0926 09:26:36.339181 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.339183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.339194 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:26:36.339225 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.339226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.339239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.339313 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:36.339666 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45888: BlockingRecv error: recv got EOF from 127.0.0.1:45888 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 09:26:36.650843 15460 catalog_manager.cc:1259] Loaded cluster ID: 371548742fc045a098aaa2634d99a6e2
I20260926 09:26:36.650914 15460 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: loading cluster ID for follower catalog manager: success
I20260926 09:26:36.651803 15460 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: acquiring CA information for follower catalog manager: success
I20260926 09:26:36.652081 15460 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8367ebc32274a0e8a720bb4a3d2d558: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:26:36.698659 15474 catalog_manager.cc:1259] Loaded cluster ID: 371548742fc045a098aaa2634d99a6e2
I20260926 09:26:36.698767 15474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: loading cluster ID for follower catalog manager: success
I20260926 09:26:36.699602 15474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: acquiring CA information for follower catalog manager: success
I20260926 09:26:36.699916 15474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f773df2e6264f0b901886ed3e7e44be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:26:36.855464 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:36.852990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.24.235.125:35443)
0926 09:26:36.853135 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.853137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.853188 (+ 51us) server_negotiation.cc:407] Connection header received
0926 09:26:36.853268 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.853268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.853323 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.853396 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:36.853519 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.854026 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.854752 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.854946 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.854997 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:36.855127 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:36.855242 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:36.855299 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:36.855319 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 09:26:36.855345 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260926 09:26:36.859375 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:36.857268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.24.235.124:43201)
0926 09:26:36.857362 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.857364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.857383 (+ 19us) server_negotiation.cc:407] Connection header received
0926 09:26:36.857418 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.857418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.857452 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.857510 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:36.857589 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.858048 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.858687 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.858885 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.858937 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:36.859043 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:36.859127 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:36.859174 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:36.859235 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 09:26:36.859251 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260926 09:26:36.863932 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:36.861168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.24.235.124:43201)
0926 09:26:36.861202 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.861203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.861382 (+ 179us) server_negotiation.cc:407] Connection header received
0926 09:26:36.861488 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.861489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.861498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.861525 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:36.861703 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.862144 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863143 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.863405 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863467 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:36.863665 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:36.863701 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:36.863782 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:36.863832 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 09:26:36.863853 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 09:26:36.864205 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:36.861087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.24.235.126:45339)
0926 09:26:36.861259 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.861261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.861272 (+ 11us) server_negotiation.cc:407] Connection header received
0926 09:26:36.861380 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.861381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.861434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.861507 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:36.861706 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.862457 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863179 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.863500 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863565 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:36.863795 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:36.863891 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:36.863948 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:36.864104 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 09:26:36.864127 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260926 09:26:36.864362 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:36.861224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.24.235.125:35443)
0926 09:26:36.861288 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:36.861290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:36.861423 (+ 133us) server_negotiation.cc:407] Connection header received
0926 09:26:36.861481 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:36.861482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:36.861494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:36.861544 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:36.861656 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.862386 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863377 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:36.863682 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:36.863758 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:36.863934 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:36.863970 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:36.863998 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:36.864271 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 09:26:36.864295 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:26:36.866055 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15225
I20260926 09:26:36.871982 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15300
I20260926 09:26:36.877345 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15375
2026-09-26T09:26:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2937 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 09:26:36 dist-test-slave-7zlw krb5kdc[15591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 09:26:36 dist-test-slave-7zlw krb5kdc[15591](info): set up 2 sockets
Sep 26 09:26:36 dist-test-slave-7zlw krb5kdc[15591](info): commencing operation
krb5kdc: starting...
W20260926 09:26:37.921674 25517 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/test-user.keytab.
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T09:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T09:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
Entry for principal kudu/127.24.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab.
I20260926 09:26:38.049952 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36057
--webserver_interface=127.24.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 09:26:38.131714 15604 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:38.131981 15604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:38.132014 15604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:38.132061 15604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:38.133275 15604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:38.133340 15604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:38.133365 15604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:38.133386 15604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:38.133412 15604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:38.134924 15604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.126.keytab
--principal=kudu/127.24.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.126:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:38.135215 15604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:38.135440 15604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:38.140558 15604 server_base.cc:1034] running on GCE node
W20260926 09:26:38.140542 15609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.140540 15610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.140533 15612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:38.140976 15604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:38.141336 15604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:38.142498 15604 hybrid_clock.cc:648] HybridClock initialized: now 1790414798142486 us; error 31 us; skew 500 ppm
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:38.144946 15604 init.cc:377] Logged in from keytab as kudu/127.24.235.126@KRBTEST.COM (short username kudu)
I20260926 09:26:38.145407 15604 webserver.cc:460] Webserver started at http://127.24.235.126:46481/ using document root <none> and password file <none>
I20260926 09:26:38.145637 15604 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:38.145699 15604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:38.145831 15604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:38.146906 15604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data/instance:
uuid: "169225c7ce7a41be8ca1b91224db0437"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "8a3f79dab3f11fa07f0eb119eb39198d"
server_key_iv: "10e2e92840023523bd59fc4dc7a409d0"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.147281 15604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance:
uuid: "169225c7ce7a41be8ca1b91224db0437"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "8a3f79dab3f11fa07f0eb119eb39198d"
server_key_iv: "10e2e92840023523bd59fc4dc7a409d0"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.148631 15604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 09:26:38.149433 15619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:38.149595 15604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:38.149698 15604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal
uuid: "169225c7ce7a41be8ca1b91224db0437"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "8a3f79dab3f11fa07f0eb119eb39198d"
server_key_iv: "10e2e92840023523bd59fc4dc7a409d0"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.149770 15604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:38.156961 15604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:38.158892 15604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:38.159047 15604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:38.163229 15604 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.126:36057
I20260926 09:26:38.163266 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.126:36057 every 8 connection(s)
I20260926 09:26:38.163626 15604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/data/info.pb
I20260926 09:26:38.164525 15672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:38.164829 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15604
I20260926 09:26:38.164927 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-0/wal/instance
I20260926 09:26:38.165169 25517 external_mini_cluster.cc:1428] Setting key a01553f099db358a55249b33c11333a7
I20260926 09:26:38.166954 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.125@KRBTEST.COM" created.
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:38.179687 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.167695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36057 (local address 127.0.0.1:51846)
0926 09:26:38.167882 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.167927 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.168480 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.168769 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.168771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.169130 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.169289 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.169294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.170278 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.170279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.170929 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.170931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.171426 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.171427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.171490 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.172023 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.172032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.176227 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.178185 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.178192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.178195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.178413 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.178663 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.178664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.178664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.178753 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.179103 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.179108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.179175 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.179270 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.179507 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":89,"thread_start_us":95,"threads_started":1}
I20260926 09:26:38.179687 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.167745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.24.235.126:36057)
0926 09:26:38.167927 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.167937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.168063 (+ 126us) server_negotiation.cc:407] Connection header received
0926 09:26:38.168637 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.168643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.168660 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.168747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.169394 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.170185 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.171033 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.171333 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.171455 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.176324 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.176332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.176333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.176341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.178080 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.178509 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.178512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.178513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.178596 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.178818 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.178819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.178819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.179034 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.179123 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.179308 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.179440 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260926 09:26:38.180639 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.181162 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.181008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54594)
0926 09:26:38.181083 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.181138 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 09:26:38.181320 15620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
I20260926 09:26:38.181766 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.181681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54596)
0926 09:26:38.181724 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.181749 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 09:26:38.181880 15672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:37051: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
Entry for principal kudu/127.24.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab.
I20260926 09:26:38.193141 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:37051
--webserver_interface=127.24.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:26:38.231118 15672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } attempt: 1
I20260926 09:26:38.231709 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.231536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54612)
0926 09:26:38.231608 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.231672 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 09:26:38.232065 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.231969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54628)
0926 09:26:38.232016 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.232042 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 09:26:38.232172 15672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:37051: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
W20260926 09:26:38.277571 15679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:38.277846 15679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:38.277876 15679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:38.277918 15679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 09:26:38.279109 15679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:38.279174 15679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:38.279198 15679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:38.279219 15679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:38.279249 15679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:38.281522 15679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.125.keytab
--principal=kudu/127.24.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.125:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:38.281826 15679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:38.282073 15679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 09:26:38.287447 15679 server_base.cc:1034] running on GCE node
W20260926 09:26:38.287431 15685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.287431 15687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.287457 15684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:38.287987 15679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:38.288296 15679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:38.289439 15679 hybrid_clock.cc:648] HybridClock initialized: now 1790414798289416 us; error 35 us; skew 500 ppm
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:38.291889 15679 init.cc:377] Logged in from keytab as kudu/127.24.235.125@KRBTEST.COM (short username kudu)
I20260926 09:26:38.292356 15679 webserver.cc:460] Webserver started at http://127.24.235.125:40301/ using document root <none> and password file <none>
I20260926 09:26:38.292599 15679 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:38.292665 15679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:38.292817 15679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:38.293969 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data/instance:
uuid: "002b4fe0e0d54025b39164fe5797ad30"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "49ed484f2611643794f12ed60552d3db"
server_key_iv: "387a7b407fbcc3e43f9a10c1bc517dbc"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.294340 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal/instance:
uuid: "002b4fe0e0d54025b39164fe5797ad30"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "49ed484f2611643794f12ed60552d3db"
server_key_iv: "387a7b407fbcc3e43f9a10c1bc517dbc"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.295755 15679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 09:26:38.296541 15694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:38.296734 15679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:38.296829 15679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal
uuid: "002b4fe0e0d54025b39164fe5797ad30"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "49ed484f2611643794f12ed60552d3db"
server_key_iv: "387a7b407fbcc3e43f9a10c1bc517dbc"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.296911 15679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:38.300328 15672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } attempt: 2
I20260926 09:26:38.300772 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.300618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54638)
0926 09:26:38.300677 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.300736 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 09:26:38.301205 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.301104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54644)
0926 09:26:38.301149 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.301180 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:26:38.301316 15672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.125:37051: Network error: Client connection negotiation failed: client connection to 127.24.235.125:37051: connect: Connection refused (error 111)
I20260926 09:26:38.307152 15679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:38.309353 15679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:38.309474 15679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:38.313968 15679 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.125:37051
I20260926 09:26:38.314033 15746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.125:37051 every 8 connection(s)
I20260926 09:26:38.314400 15679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/data/info.pb
I20260926 09:26:38.315285 15747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:38.316845 15747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.318298 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15679
I20260926 09:26:38.318413 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-1/wal/instance
I20260926 09:26:38.318645 25517 external_mini_cluster.cc:1428] Setting key 63c762650c3b4e1dbedb04fc2f78f9f1
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:38.326676 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.317588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.24.235.126:36057)
0926 09:26:38.317639 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.317641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.317920 (+ 279us) server_negotiation.cc:407] Connection header received
0926 09:26:38.318414 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.318415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.318431 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.318494 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.319167 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.319665 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.320656 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.320877 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.320963 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.323465 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.323467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.323468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.323476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.325183 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.325607 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.325608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.325609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.325670 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.325953 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.325954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.325955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.326110 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.326192 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.326479 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.326570 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.24.235.124@KRBTEST.COM; defaulting to no policy
I20260926 09:26:38.326767 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.317575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36057 (local address 127.0.0.1:51852)
0926 09:26:38.317798 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.317833 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.318305 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.318568 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.318571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.318877 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.319074 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.319078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.319769 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.319770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.320563 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.320564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.320976 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.320977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.321130 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.321689 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.321704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.323365 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.325265 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.325271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.325274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.325529 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.325764 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.325765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.325765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.325879 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.326193 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.326217 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.326342 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.326433 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.326560 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260926 09:26:38.327338 15747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.235.124@KRBTEST.COM" created.
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:38.335098 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.327604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54650)
0926 09:26:38.327651 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.327662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.327736 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.328051 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.328052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.328192 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.328242 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.328243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.329272 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.329272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.329816 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.329818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.330293 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.330294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.330357 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.330808 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.330812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.331919 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.333914 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.333915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.333915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.334100 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.334332 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.334333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.334333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.334392 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.334727 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.334728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.334769 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.334853 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.334963 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 09:26:38.335126 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.327690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.24.235.125:37051)
0926 09:26:38.327920 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.327924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.327940 (+ 16us) server_negotiation.cc:407] Connection header received
0926 09:26:38.327958 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.327959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.327969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.328052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.328331 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.329195 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.329903 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.330210 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.330294 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.332011 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.332019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.332019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.332026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.333797 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.334199 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.334202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.334202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.334261 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.334463 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.334464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.334465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.334641 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.334734 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.334887 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.335020 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260926 09:26:38.335969 15747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.336374 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.336234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58564)
0926 09:26:38.336286 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.336340 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 09:26:38.336540 15696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
I20260926 09:26:38.337064 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.336931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58578)
0926 09:26:38.336977 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.337004 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 09:26:38.337157 15747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43615: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
Entry for principal kudu/127.24.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab.
I20260926 09:26:38.347630 25517 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskr9gC17/build/release/bin/kudu
/tmp/dist-test-taskr9gC17/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:43615
--webserver_interface=127.24.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.235.84:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 09:26:38.386406 15747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } attempt: 1
I20260926 09:26:38.387048 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.386906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58592)
0926 09:26:38.386971 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.387016 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 09:26:38.387442 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.387341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58602)
0926 09:26:38.387385 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.387419 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 09:26:38.387542 15747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43615: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
I20260926 09:26:38.392452 15672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } attempt: 3
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:38.399657 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.392842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.24.235.125:37051)
0926 09:26:38.392910 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.392912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.392924 (+ 12us) server_negotiation.cc:407] Connection header received
0926 09:26:38.392963 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.392964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.392982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.393056 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.393395 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.393886 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.394577 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.394798 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.394874 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.396824 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.396825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.396825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.396833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.398314 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.398721 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.398722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.398723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.398785 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.398998 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.398999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.398999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.399139 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.399240 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.399404 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.399494 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 09:26:38.399674 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.392753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54666)
0926 09:26:38.392811 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.392824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.392896 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.393074 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.393075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.393246 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.393314 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.393315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.393967 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.393968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.394493 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.394495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.394871 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.394872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.394953 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.395440 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.395444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.396741 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.398406 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.398408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.398408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.398629 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.398864 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.398865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.398865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.398931 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.399231 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.399232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.399286 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.399378 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.399492 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 09:26:38.400216 15672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.400650 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.400508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58610)
0926 09:26:38.400582 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.400620 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 09:26:38.401072 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.400931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58620)
0926 09:26:38.400980 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.401033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 09:26:38.401190 15672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43615: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
W20260926 09:26:38.431953 15754 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 09:26:38.432178 15754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 09:26:38.432196 15754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 09:26:38.432217 15754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 09:26:38.432350 15672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } attempt: 1
I20260926 09:26:38.432809 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.432652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58624)
0926 09:26:38.432723 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.432772 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 09:26:38.433199 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.433105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58628)
0926 09:26:38.433147 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.433177 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 09:26:38.433295 15672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43615: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
W20260926 09:26:38.434089 15754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 09:26:38.434139 15754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 09:26:38.434166 15754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 09:26:38.434190 15754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 09:26:38.434212 15754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 09:26:38.435798 15754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.235.84:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.235.126:36057,127.24.235.125:37051,127.24.235.124:43615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/krb5kdc/kudu_127.24.235.124.keytab
--principal=kudu/127.24.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.235.124:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 08:27:02 UTC on ccycloud.quasar-klbodp.root.comops.site
build id 4673776
I20260926 09:26:38.436064 15754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 09:26:38.436296 15754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 09:26:38.441219 15760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.441219 15759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 09:26:38.441294 15762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 09:26:38.441541 15754 server_base.cc:1034] running on GCE node
I20260926 09:26:38.441784 15754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 09:26:38.442067 15754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 09:26:38.443216 15754 hybrid_clock.cc:648] HybridClock initialized: now 1790414798443185 us; error 51 us; skew 500 ppm
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 09:26:38.445919 15754 init.cc:377] Logged in from keytab as kudu/127.24.235.124@KRBTEST.COM (short username kudu)
I20260926 09:26:38.446421 15754 webserver.cc:460] Webserver started at http://127.24.235.124:37433/ using document root <none> and password file <none>
I20260926 09:26:38.446653 15754 fs_manager.cc:359] Metadata directory not provided
I20260926 09:26:38.446699 15754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 09:26:38.446846 15754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 09:26:38.447925 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data/instance:
uuid: "bf6fd690664b4aefbb135703bb1a2889"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "24ccdde2e11f72a1ed96d11635c744f5"
server_key_iv: "0c556e0f9b425096f47ecaa7b3410839"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.448307 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal/instance:
uuid: "bf6fd690664b4aefbb135703bb1a2889"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "24ccdde2e11f72a1ed96d11635c744f5"
server_key_iv: "0c556e0f9b425096f47ecaa7b3410839"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.449787 15754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 09:26:38.450691 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 09:26:38.450940 15754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 09:26:38.451051 15754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data,/tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal
uuid: "bf6fd690664b4aefbb135703bb1a2889"
format_stamp: "Formatted at 2026-09-26 09:26:38 on dist-test-slave-7zlw"
server_key: "24ccdde2e11f72a1ed96d11635c744f5"
server_key_iv: "0c556e0f9b425096f47ecaa7b3410839"
server_key_version: "encryptionkey@0"
I20260926 09:26:38.451151 15754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 09:26:38.455677 15747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } attempt: 2
I20260926 09:26:38.456115 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.456008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58636)
0926 09:26:38.456059 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.456087 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 09:26:38.456465 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.456379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58652)
0926 09:26:38.456417 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.456441 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 09:26:38.456545 15747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.235.124:43615: Network error: Client connection negotiation failed: client connection to 127.24.235.124:43615: connect: Connection refused (error 111)
I20260926 09:26:38.461783 15754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 09:26:38.464146 15754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 09:26:38.464316 15754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 09:26:38.468482 15754 rpc_server.cc:307] RPC server started. Bound to: 127.24.235.124:43615
I20260926 09:26:38.468513 15821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.235.124:43615 every 8 connection(s)
I20260926 09:26:38.468853 15754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/data/info.pb
I20260926 09:26:38.469775 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 09:26:38.471321 15822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.472993 25517 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskr9gC17/build/release/bin/kudu as pid 15754
I20260926 09:26:38.473145 25517 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskr9gC17/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790414522268193-25517-0/minicluster-data/master-2/wal/instance
I20260926 09:26:38.473357 25517 external_mini_cluster.cc:1428] Setting key 0ee6f7c8cb35588bc7bcfb3c1fed6edf
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:38.481199 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.472059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.24.235.126:36057)
0926 09:26:38.472125 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.472127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.472312 (+ 185us) server_negotiation.cc:407] Connection header received
0926 09:26:38.472906 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.472908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.472920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.473004 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.473938 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.474448 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.475434 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.475625 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.475706 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.478062 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.478064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.478065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.478073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.479724 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.480216 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.480218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.480219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.480280 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.480520 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.480521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.480521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.480664 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.480742 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.480985 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.481107 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 09:26:38.481417 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.471991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.126:36057 (local address 127.0.0.1:51866)
0926 09:26:38.472192 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.472234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.472711 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.473126 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.473131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.473451 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.473793 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.473799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.474552 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.474553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.475340 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.475342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.475715 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.475716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.475844 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.476424 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.476434 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.477973 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.479802 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.479809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.479810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.480119 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.480348 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.480349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.480349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.480454 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.480737 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.480741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.480859 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.480941 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.481130 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":17,"thread_start_us":113,"threads_started":1}
I20260926 09:26:38.482100 15822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.482494 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.473727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.24.235.124:43615)
0926 09:26:38.473888 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.473893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.473923 (+ 30us) server_negotiation.cc:407] Connection header received
0926 09:26:38.473989 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.473992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.474013 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.474097 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.474724 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.475467 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.476231 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.476676 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.476758 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.479236 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.479244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.479244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.479251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.481012 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.481394 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.481397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.481398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.481455 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.481697 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.481698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.481698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.481899 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.482066 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.482114 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.482192 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.126@KRBTEST.COM
I20260926 09:26:38.489419 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.482439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.24.235.125:37051)
0926 09:26:38.482477 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.482479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.482499 (+ 20us) server_negotiation.cc:407] Connection header received
0926 09:26:38.482530 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.482531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.482539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.482600 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.482899 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.483583 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.484343 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.484570 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.484652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.486601 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.486603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.486603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.486610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.488130 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.488538 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.488539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.488540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.488606 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.488847 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.488849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.488849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.488967 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.489133 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.489291 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.489355 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:26:38.489480 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.482363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.125:37051 (local address 127.0.0.1:54676)
0926 09:26:38.482412 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.482422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.482458 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.482623 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.482624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.482759 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.482812 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.482814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.483665 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.483666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.484261 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.484263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.484653 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.484654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.484724 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.485137 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.485142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.486521 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.488212 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.488213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.488213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.488463 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.488682 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.488683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.488683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.488750 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.489130 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.489131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.489181 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.489262 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.489358 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 09:26:38.489991 15822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
I20260926 09:26:38.491019 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.483262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.24.235.126:36057)
0926 09:26:38.483298 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.483300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.483350 (+ 50us) server_negotiation.cc:407] Connection header received
0926 09:26:38.483421 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.483422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.483431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.483478 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.483915 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.484455 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.485142 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.485338 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.485407 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.487623 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.487624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.487625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.487632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.489416 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.490058 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.490059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.490059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.490135 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.490402 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.490403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.490404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.490532 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.490742 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.490824 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.490910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.124@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.235.125@KRBTEST.COM
I20260926 09:26:38.497068 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.490319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58680)
0926 09:26:38.490354 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.490365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.490403 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.490523 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.490524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.490656 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.490682 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.490683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.491430 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.491431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.492063 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.492066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.492494 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.492495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.492555 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.492896 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.492901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.494172 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.496097 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.496097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.496098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.496256 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.496456 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.496456 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.496457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.496510 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.496750 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.496751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.496787 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.496850 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.496949 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 09:26:38.497159 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.490360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.24.235.124:43615)
0926 09:26:38.490394 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.490396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.490404 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:26:38.490458 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.490459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.490469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.490520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.490753 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.491351 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.492164 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.492420 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.492468 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.494274 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.494275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.494276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.494282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.496026 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.496338 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.496339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.496340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.496387 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.496575 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.496575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.496576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.496687 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.496757 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.496876 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.496996 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":20}
I20260926 09:26:38.498761 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: Bootstrap starting.
I20260926 09:26:38.499403 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.491653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.24.235.125:37051)
0926 09:26:38.491695 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.491697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.491705 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:26:38.491727 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.491728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.491737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.491790 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.492252 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.492932 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.493662 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.493869 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.493944 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.496394 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.496396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.496396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.496404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.498228 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.498571 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.498572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.498573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.498625 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.498820 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.498821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.498822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.498943 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.499150 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.499188 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.499286 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:38.499826 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:38.500180 15822 log.cc:824] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:38.500988 25517 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 09:26:38.501128 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: No bootstrap required, opened a new log
I20260926 09:26:38.502157 15646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "169225c7ce7a41be8ca1b91224db0437"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51874
I20260926 09:26:38.502861 15822 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.503026 15822 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6fd690664b4aefbb135703bb1a2889, State: Initialized, Role: FOLLOWER
I20260926 09:26:38.503139 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.503491 15822 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:38.503465 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.503563 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 09:26:38.505272 15839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:38.505316 15839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:38.505756 15754 master_runner.cc:186] Error getting master registration for 127.24.235.126:36057: OK, instance_id { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" instance_seqno: 1790414798162032 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:26:38.507820 15754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:26:38.508425 15672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } attempt: 2
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.126@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:38.515440 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.508727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58682)
0926 09:26:38.508786 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.508800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.508853 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.508981 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.508982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.509184 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.509227 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.509229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.509853 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.509855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.510355 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.510357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.510726 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.510727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.510785 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.511225 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.511235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.512389 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.514276 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.514277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.514278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.514506 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.514724 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.514725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.514725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.514789 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.515110 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.515111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.515160 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.515232 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.515333 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 09:26:38.515448 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.508804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.24.235.124:43615)
0926 09:26:38.508867 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.508869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.508879 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:38.508904 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.508905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.508915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.508977 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.509314 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.509779 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.510437 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.510643 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.510712 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.512472 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.512473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.512474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.512482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.514194 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.514583 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.514585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.514585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.514647 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.514853 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.514854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.514855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.515037 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.515104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.515276 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.515356 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 09:26:38.517475 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: Bootstrap starting.
I20260926 09:26:38.518126 15672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:38.518427 15672 log.cc:824] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:38.519284 15672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: No bootstrap required, opened a new log
I20260926 09:26:38.520309 15672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.520448 15672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169225c7ce7a41be8ca1b91224db0437, State: Initialized, Role: FOLLOWER
I20260926 09:26:38.520543 15672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.520882 15672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:38.521034 15646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 09:26:38.521106 15646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:38.521876 15646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.522114 15646 leader_election.cc:288] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [CANDIDATE]: Term 1 election: Requested vote from peers 002b4fe0e0d54025b39164fe5797ad30 (127.24.235.125:37051), bf6fd690664b4aefbb135703bb1a2889 (127.24.235.124:43615)
I20260926 09:26:38.522698 15796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "169225c7ce7a41be8ca1b91224db0437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf6fd690664b4aefbb135703bb1a2889"
I20260926 09:26:38.522826 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [term 0 FOLLOWER]: Advancing to term 1
W20260926 09:26:38.523103 15853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:38.523155 15853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:38.523699 15796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169225c7ce7a41be8ca1b91224db0437 in term 1.
I20260926 09:26:38.524818 15621 leader_election.cc:302] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169225c7ce7a41be8ca1b91224db0437, bf6fd690664b4aefbb135703bb1a2889; no voters:
I20260926 09:26:38.525214 15862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 09:26:38.525394 15862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [term 1 LEADER]: Becoming Leader. State: Replica: 169225c7ce7a41be8ca1b91224db0437, State: Running, Role: LEADER
I20260926 09:26:38.525480 15862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.525171 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.525759 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.526127 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169225c7ce7a41be8ca1b91224db0437. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.526192 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:38.526834 15865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 09:26:38.527010 15865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 09:26:38.525158 15604 master_runner.cc:186] Error getting master registration for 127.24.235.125:37051: OK, instance_id { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" instance_seqno: 1790414798312558 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 09:26:38.527520 15604 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 09:26:38.539361 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.523763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.24.235.126:36057)
0926 09:26:38.523807 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.523810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.527402 (+ 3592us) server_negotiation.cc:407] Connection header received
0926 09:26:38.528086 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.528087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.528097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.528125 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.529439 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.530197 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.532965 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.533282 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.533324 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.536245 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.536246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.536247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.536254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.537976 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.538363 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.538364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.538364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.538418 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.538612 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.538613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.538613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.538764 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.538848 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.539186 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.539261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 09:26:38.540539 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.524700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.24.235.124:43615)
0926 09:26:38.524742 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.524744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.527494 (+ 2750us) server_negotiation.cc:407] Connection header received
0926 09:26:38.527879 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.527880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.527892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.527923 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.530274 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.530998 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.533406 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.533722 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.533761 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.536244 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.536246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.536246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.536254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.538744 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.539750 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.539751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.539751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.539809 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.539992 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.539992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.539993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.540113 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.540191 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.540373 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.540448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 09:26:38.540680 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.524475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.24.235.125:37051)
0926 09:26:38.524532 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.524534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.527582 (+ 3048us) server_negotiation.cc:407] Connection header received
0926 09:26:38.527868 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.527869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.527884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.527947 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.530055 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.530817 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.532429 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.533289 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.533360 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.535878 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.535879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.535880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.535904 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.537974 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.539329 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.539331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.539331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.539401 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.539985 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.539986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.539987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.540120 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.540203 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.540506 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.540590 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 09:26:38.541939 15796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [term 1 FOLLOWER]: Refusing update from remote peer 169225c7ce7a41be8ca1b91224db0437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:38.542392 15842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:38.543778 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169225c7ce7a41be8ca1b91224db0437. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.543882 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.545380 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.545466 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:38.545544 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.545634 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.545804 15867 mvcc.cc:205] Tried to move back new op lower bound from 7333539014825418752 to 7333539014762020864. Current Snapshot: MvccSnapshot[applied={T|T < 7333539014825418752}]
I20260926 09:26:38.547397 15865 catalog_manager.cc:1347] Generated new cluster ID: d60ea9088dea4381b221e2684536b40f
I20260926 09:26:38.547447 15865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 09:26:38.547694 15747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } attempt: 3
Sep 26 09:26:38 dist-test-slave-7zlw krb5kdc[15591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790414798, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.235.125@KRBTEST.COM for kudu/127.24.235.124@KRBTEST.COM
I20260926 09:26:38.556797 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.547922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.235.124:43615 (local address 127.0.0.1:58698)
0926 09:26:38.547999 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 09:26:38.548009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 09:26:38.548046 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 09:26:38.548302 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 09:26:38.548303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 09:26:38.548462 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 09:26:38.548507 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.548509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.549465 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.549466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.550265 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 09:26:38.550267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.550761 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.550762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 09:26:38.550817 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.551266 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 09:26:38.551272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 09:26:38.553436 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 09:26:38.555512 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.555513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.555513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.555745 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.555976 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 09:26:38.555977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 09:26:38.555977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 09:26:38.556044 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 09:26:38.556465 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 09:26:38.556466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 09:26:38.556523 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 09:26:38.556584 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 09:26:38.556693 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 09:26:38.557852 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:38.548125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.24.235.124:43615)
0926 09:26:38.548158 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.548160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.548168 (+ 8us) server_negotiation.cc:407] Connection header received
0926 09:26:38.548187 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.548188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.548197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.548242 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.548586 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.549374 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.550354 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:38.550643 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:38.550710 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:38.553524 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 09:26:38.553526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 09:26:38.553526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 09:26:38.553533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 09:26:38.555346 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.555834 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.555835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.555835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.555901 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 09:26:38.556117 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 09:26:38.556140 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 09:26:38.556140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 09:26:38.556358 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 09:26:38.556429 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:38.557681 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0926 09:26:38.557762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:26:38.559455 15747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: Bootstrap starting.
I20260926 09:26:38.560209 15747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: Neither blocks nor log segments found. Creating new log.
I20260926 09:26:38.560523 15747 log.cc:824] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: Log is configured to *not* fsync() on all Append() calls
I20260926 09:26:38.573860 15747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: No bootstrap required, opened a new log
I20260926 09:26:38.576201 15747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.576376 15747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002b4fe0e0d54025b39164fe5797ad30, State: Initialized, Role: FOLLOWER
I20260926 09:26:38.576473 15747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } }
I20260926 09:26:38.576851 15747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 09:26:38.576933 15720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 09:26:38.576944 15721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "169225c7ce7a41be8ca1b91224db0437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "002b4fe0e0d54025b39164fe5797ad30"
I20260926 09:26:38.578058 15720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [term 1 FOLLOWER]: Refusing update from remote peer 169225c7ce7a41be8ca1b91224db0437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 09:26:38.578420 15862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 09:26:38.579183 15721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169225c7ce7a41be8ca1b91224db0437 in term 1.
W20260926 09:26:38.582044 15883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 09:26:38.582680 15883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 09:26:38.582837 15865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 09:26:38.582629 15885 mvcc.cc:205] Tried to move back new op lower bound from 7333539014825418752 to 7333539014762020864. Current Snapshot: MvccSnapshot[applied={T|T < 7333539014825418752}]
I20260926 09:26:38.582224 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.583901 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.583930 15865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 09:26:38.584012 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.584125 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.584326 15887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 169225c7ce7a41be8ca1b91224db0437. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.584450 15887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 09:26:38.584870 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "169225c7ce7a41be8ca1b91224db0437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169225c7ce7a41be8ca1b91224db0437" member_type: VOTER last_known_addr { host: "127.24.235.126" port: 36057 } } peers { permanent_uuid: "002b4fe0e0d54025b39164fe5797ad30" member_type: VOTER last_known_addr { host: "127.24.235.125" port: 37051 } } peers { permanent_uuid: "bf6fd690664b4aefbb135703bb1a2889" member_type: VOTER last_known_addr { host: "127.24.235.124" port: 43615 } } }
I20260926 09:26:38.584954 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437 [sys.catalog]: This master's current role is: LEADER
I20260926 09:26:38.601459 15865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 169225c7ce7a41be8ca1b91224db0437: Generated new TSK 0
I20260926 09:26:38.601696 15865 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 09:26:38.628556 15676 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.626759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.24.235.126:36057)
0926 09:26:38.626808 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.626810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.627831 (+ 1021us) server_negotiation.cc:407] Connection header received
0926 09:26:38.627942 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.627943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.627956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.627987 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.628480 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51884: BlockingRecv error: recv got EOF from 127.0.0.1:51884 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 09:26:38.629559 15826 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.627173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.24.235.124:43615)
0926 09:26:38.627219 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.627220 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.629031 (+ 1811us) server_negotiation.cc:407] Connection header received
0926 09:26:38.629136 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.629137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.629149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.629178 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.629494 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58714: BlockingRecv error: recv got EOF from 127.0.0.1:58714 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 09:26:38.630422 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.627540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.24.235.125:37051)
0926 09:26:38.627581 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.627583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.629918 (+ 2335us) server_negotiation.cc:407] Connection header received
0926 09:26:38.630015 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.630016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.630027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.630056 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.630362 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54696: BlockingRecv error: recv got EOF from 127.0.0.1:54696 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:26:38.632782 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.632074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.24.235.125:37051)
0926 09:26:38.632109 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.632111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.632279 (+ 168us) server_negotiation.cc:407] Connection header received
0926 09:26:38.632382 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.632383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.632401 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.632430 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.632739 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54700: BlockingRecv error: recv got EOF from 127.0.0.1:54700 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:26:38.635932 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.635176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.24.235.125:37051)
0926 09:26:38.635223 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.635225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.635435 (+ 210us) server_negotiation.cc:407] Connection header received
0926 09:26:38.635530 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.635531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.635541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.635569 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.635888 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54712: BlockingRecv error: recv got EOF from 127.0.0.1:54712 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 09:26:38.639918 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.639166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.24.235.125:37051)
0926 09:26:38.639200 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.639202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.639415 (+ 213us) server_negotiation.cc:407] Connection header received
0926 09:26:38.639510 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.639510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.639521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.639549 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.639875 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54720: BlockingRecv error: recv got EOF from 127.0.0.1:54720 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 09:26:38.646113 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.645275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.24.235.125:37051)
0926 09:26:38.645313 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.645315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.645559 (+ 244us) server_negotiation.cc:407] Connection header received
0926 09:26:38.645660 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.645661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.645672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.645717 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.646064 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54736: BlockingRecv error: recv got EOF from 127.0.0.1:54736 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:26:38.656417 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.655540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.24.235.125:37051)
0926 09:26:38.655582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.655584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.655834 (+ 250us) server_negotiation.cc:407] Connection header received
0926 09:26:38.655939 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.655941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.655954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.655984 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.656337 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54738: BlockingRecv error: recv got EOF from 127.0.0.1:54738 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 09:26:38.675041 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.674067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.24.235.125:37051)
0926 09:26:38.674118 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.674120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.674388 (+ 268us) server_negotiation.cc:407] Connection header received
0926 09:26:38.674497 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.674498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.674511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.674544 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.674976 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54742: BlockingRecv error: recv got EOF from 127.0.0.1:54742 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 09:26:38.709380 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.708535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.24.235.125:37051)
0926 09:26:38.708585 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.708587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.708753 (+ 166us) server_negotiation.cc:407] Connection header received
0926 09:26:38.708833 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.708835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.708848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.708917 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.709304 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54756: BlockingRecv error: recv got EOF from 127.0.0.1:54756 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 09:26:38.775624 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.774786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.24.235.125:37051)
0926 09:26:38.774882 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.774884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.774932 (+ 48us) server_negotiation.cc:407] Connection header received
0926 09:26:38.775024 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.775025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.775037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.775087 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.775469 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54766: BlockingRecv error: recv got EOF from 127.0.0.1:54766 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 09:26:38.905771 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:38.904965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.24.235.125:37051)
0926 09:26:38.905049 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:38.905051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:38.905137 (+ 86us) server_negotiation.cc:407] Connection header received
0926 09:26:38.905174 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:38.905175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:38.905187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:38.905297 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:38.905663 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54776: BlockingRecv error: recv got EOF from 127.0.0.1:54776 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 09:26:39.163825 15751 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 09:26:39.163019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.24.235.125:37051)
0926 09:26:39.163103 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.163105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.163154 (+ 49us) server_negotiation.cc:407] Connection header received
0926 09:26:39.163256 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.163257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.163270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.163339 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 09:26:39.163774 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54782: BlockingRecv error: recv got EOF from 127.0.0.1:54782 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 09:26:39.505825 15839 catalog_manager.cc:1259] Loaded cluster ID: d60ea9088dea4381b221e2684536b40f
I20260926 09:26:39.505936 15839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: loading cluster ID for follower catalog manager: success
I20260926 09:26:39.506795 15839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: acquiring CA information for follower catalog manager: success
I20260926 09:26:39.507103 15839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf6fd690664b4aefbb135703bb1a2889: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:26:39.583243 15883 catalog_manager.cc:1259] Loaded cluster ID: d60ea9088dea4381b221e2684536b40f
I20260926 09:26:39.583356 15883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: loading cluster ID for follower catalog manager: success
I20260926 09:26:39.584210 15883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: acquiring CA information for follower catalog manager: success
I20260926 09:26:39.584537 15883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 002b4fe0e0d54025b39164fe5797ad30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 09:26:39.679992 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:39.677043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.24.235.125:37051)
0926 09:26:39.677184 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.677186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.677251 (+ 65us) server_negotiation.cc:407] Connection header received
0926 09:26:39.677367 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.677368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.677426 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.677503 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:39.677635 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.678132 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.679026 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.679229 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.679285 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:39.679475 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:39.679596 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:39.679655 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:39.679827 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 09:26:39.679851 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260926 09:26:39.684429 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:39.681964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.24.235.124:43615)
0926 09:26:39.682097 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.682099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.682109 (+ 10us) server_negotiation.cc:407] Connection header received
0926 09:26:39.682149 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.682149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.682187 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.682265 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:39.682416 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.682933 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.683667 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.683876 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.683933 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:39.684083 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:39.684182 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:39.684243 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:39.684278 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 09:26:39.684297 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260926 09:26:39.688858 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:39.686404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.24.235.124:43615)
0926 09:26:39.686441 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.686443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.686538 (+ 95us) server_negotiation.cc:407] Connection header received
0926 09:26:39.686626 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.686626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.686636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.686678 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:39.686813 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.687379 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688123 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.688333 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688413 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:39.688565 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:39.688605 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:39.688662 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:39.688728 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 09:26:39.688749 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 09:26:39.689379 15955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:39.686142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.24.235.125:37051)
0926 09:26:39.686180 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.686181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.686286 (+ 105us) server_negotiation.cc:407] Connection header received
0926 09:26:39.686388 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.686388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.686400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.686429 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:39.686612 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.687397 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688577 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.688877 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688916 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:39.689131 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:39.689160 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:39.689205 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:39.689280 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 09:26:39.689305 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 09:26:39.689955 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 09:26:39.686184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.24.235.126:36057)
0926 09:26:39.686289 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 09:26:39.686291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 09:26:39.686300 (+ 9us) server_negotiation.cc:407] Connection header received
0926 09:26:39.686377 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 09:26:39.686377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 09:26:39.686430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 09:26:39.686531 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 09:26:39.686657 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.687422 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688575 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 09:26:39.688882 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 09:26:39.688948 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 09:26:39.689536 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 09:26:39.689665 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 09:26:39.689739 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 09:26:39.689824 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 09:26:39.689848 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260926 09:26:39.696550 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15604
I20260926 09:26:39.702960 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15679
I20260926 09:26:39.708446 25517 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskr9gC17/build/release/bin/kudu with pid 15754
2026-09-26T09:26:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2831 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5769 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277442 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST